.. include:: ../vars.rst

.. _chapter-observability-framework:

***************************************
Support for RTI |OBSERVABILITY_HEADING|
***************************************

This chapter demonstrates how to configure the |RTI_SP_PRODUCT| to secure 
DDS communications between |RTI_MONITORINGLIBRARY2| and |RTI_COLLECTORSERVICE|, 
or between a |COLLECTORSERVICE| instance with a DDS exporter acting as a 
forwarder and a |COLLECTORSERVICE| instance with a DDS receiver. For information 
about configuration, usage, and components of |OBSERVABILITY|, refer to the
:link_addon_products_observability:`RTI Connext Observability Framework User's Manual <index.html>`. 

.. attention::

    If your |CONNEXT| application uses |MONITORINGLIBRARY2|, then you **must
    secure** the DDS communication between |MONITORINGLIBRARY2| and
    |COLLECTORSERVICE| using the |RTI_SP_PRODUCT| as explained in this section.
    Otherwise, the enablement of |MONITORINGLIBRARY2| will fail with the
    following error message:

    .. code-block:: text

        [ERROR] FAILED TO ENABLE | You are attempting to enable the RTI Monitoring Library 2.0 without security, but your system already contains a DomainParticipant with security enabled. Please, enable security for the RTI Monitoring Library 2.0 (refer to the RTI Connext Observability Framework for further information).

    For the same reason, if your application has already enabled |MONITORINGLIBRARY2|
    without security and you later enable a |DP| using the |RTI_SP_PRODUCT|
    or |LIGHT_SP_BUILTIN|, the |DP| enablement will fail with the following
    error message:

    .. code-block:: text

        [ERROR] FAILED TO ENABLE | You are attempting to enable a secure DomainParticipant, but the RTI Monitoring Library 2.0 is not secured. Please, disable RTI Monitoring Library 2.0 (using factoryQos.monitoring.enable) or enable security for it (refer to the RTI Connext Observability Framework for further information).

Security for Telemetry Data
===========================

If your |CONNEXT| applications use |OBSERVABILITY|, you may want to
secure the DDS traffic published by |MONITORINGLIBRARY2| and control which
|COLLECTORSERVICE| can subscribe to it.
:link_addon_products_observability:`Telemetry data <whatis.html#telemetry-data>`
may contain sensitive information about your system (for example, logging
messages) that must be protected from unauthorized access.

|MONITORINGLIBRARY2| uses a |DP|, |TOPICs|, and |DWs| to publish telemetry data.
|COLLECTORSERVICE| uses a |DP|, |TOPICs|, and |DRs| to subscribe to
that data. Therefore, the |RTI_SP_PRODUCT| can be used to secure the communications between them as if
they were regular |CONNEXT| applications. The |GovernanceDoc| and the |PermissionsDoc|
provide specific XML tags for Observability.

Configuring security for telemetry data does not affect your application data.
You can secure telemetry data independently from your application data and vice
versa.

.. important::
  * Refer to :ref:`section-recommendations-for-usage-with-observability` for
    suggestions on how to configure the |RTI_SP_PRODUCT| for |OBSERVABILITY|.
  * |LIGHT_SP_BUILTIN| **is not supported** for |OBSERVABILITY| evaluation
    deployments.

Securing Telemetry Data with Security Plugins
---------------------------------------------


Creating a |GovernanceDoc| for |OBSERVABILITY_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The first thing you need to determine is the level of protection you want for
telemetry data (:xmlval:`ENCRYPT`, :xmlval:`ENCRYPT_WITH_ORIGIN_AUTHENTICATION`,
:xmlval:`SIGN`, :xmlval:`SIGN_WITH_ORIGIN_AUTHENTICATION`, or :xmlval:`NONE`).
Metrics and logs can have different levels of sensitivity, so you may want to
protect them differently. Use the :xmltag:`monitoring_metrics_protection_kind`
and :xmltag:`monitoring_logging_protection_kind` tags to set the protection
level for metrics and logs, respectively. For example, if you want to encrypt
logs but not metrics:

.. literalinclude:: ../snippets/observability_governance_file_sign_metrics_encrypt_logs.xml
  :language: xml
  :emphasize-lines: 10-11, 17-18
  :caption: Example |GovernanceDoc| for |OBSERVABILITY|

.. note::
  :xmltag:`monitoring_metrics_protection_kind` applies to both event and
  periodic metrics.

.. note::
  By default, |OBSERVABILITY| uses the domain tag ``RTI_o11y`` for the
  observability domain. This is why the example |GovernanceDoc| includes
  :xmlval:`<tag>RTI_o11y</tag>` in the :xmltag:`domains` section—it matches
  the observability |DP|.

  The empty :xmlval:`<tag></tag>` is only needed if this domain rule
  is shared with the application domains, which by default do not have a
  domain tag. The empty tag matches |DPs| that have no domain tag configured.
  If the |GovernanceDoc| is not shared with application domains, you do not
  need to include the empty tag.

If one of these tags is not present in the |GovernanceDoc|, the
protection level for that data (metrics or logs) is inherited from 
:xmltag:`discovery_protection_kind`. Both tags are optional.

.. attention::
  Protection kinds for Observability |TOPICs| must be configured using
  the XML tags described in this section. If they are configured using
  the |TOPIC| names in the :xmltag:`topic_expression` tag, an error will occur. 
  
  Using wildcards will not result in an error. However, even if the wildcard 
  matches a |TOPIC| name, the rule will not be applied.

.. important::
  If you also want to secure remote administration commands, you need to protect
  the ServiceRequest channel using the :xmltag:`service_request_protection_kind`
  XML tag. See :ref:`section-service-request-protection-kind` for further
  information.

Creating a |PermissionsDoc| for |COLLECTORSERVICE_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The second thing you need to determine is which *Collector Services* can
subscribe to telemetry data and which data they can subscribe to. For 
example, you may want to allow one |COLLECTORSERVICE| to subscribe to 
all telemetry data, while another subscribes only to metrics. Use the 
:xmltag:`subscribe_monitoring` XML tag in the |PermissionsDoc| to set which 
data a |COLLECTORSERVICE| instance can subscribe to. In the following 
example |PermissionsDoc|, the |COLLECTORSERVICE| subscribes to all 
telemetry data:

.. literalinclude:: ../snippets/observability_permissions_file_subscribe_monitoring_all.xml
  :language: xml
  :emphasize-lines: 13-14, 16
  :caption: Example |PermissionsDoc| for |COLLECTORSERVICE|

.. note::
  By default, |OBSERVABILITY| uses domain ID 101 with a domain tag of
  ``RTI_o11y``. This is why the allow rule in the example |PermissionsDoc|
  includes :xmlval:`<id>101</id>` and :xmlval:`<tag>RTI_o11y</tag>` in the
  :xmltag:`domains` section. If you change the observability domain ID or
  domain tag, update the :xmltag:`domains` in the allow rule accordingly.

Valid values for :xmltag:`subscribe_monitoring` are :xmlval:`ALL`
(logging and metrics), :xmlval:`METRICS`, and :xmlval:`NONE`. Note that a
|COLLECTORSERVICE| cannot subscribe to logging data only. Also, this tag can
only be used in allow rules.

.. attention::
  * The default rule for |OBSERVABILITY| endpoints is always :xmlval:`DENY`
    regardless of the value configured in the |PermissionsDoc|.

  * The :xmlval:`METRICS` value for the :xmltag:`subscribe_monitoring` tag
    **is not currently supported**.

  * Observability endpoint permissions must be configured using
    the :xmltag:`subscribe_monitoring` XML tag. If these permissions are 
    configured using the |TOPIC| names in :xmltag:`topics`, an error will occur. 
    Using wildcards will not result in an error. However, even if the wildcard
    matches a |TOPIC| name, the rule will not be applied.

Allow rules in the |PermissionsDoc| are evaluated in order. If the Observability
Domain ID matches the allowed :xmltag:`domains`, the |RTI_SP_PRODUCT| will check the
:xmltag:`subscribe_monitoring` tag in that allow rule. The rule will be ignored,
and the next rule will be evaluated, if the tag is not present, if the tag is set
to :xmlval:`NONE`, or if the current |TOPIC| is the logging |TOPIC| and the tag
is set to :xmlval:`METRICS`. The process continues until a rule applies. If none
of the allow rules apply, the default rule will be applied (always
:xmlval:`DENY`).

Creating a |PermissionsDoc| for |MONITORINGLIBRARY2_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Publishing telemetry data **is always allowed**. Therefore, you do not need to
create a special |PermissionsDoc| for |MONITORINGLIBRARY2|. You can reuse the
same |PermissionsDoc| you created for |COLLECTORSERVICE|
(although :xmltag:`subscribe_monitoring` is not required in this case).

Enabling |RTI_SP_PRODUCT_HEADING| in |COLLECTORSERVICE_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

The enablement of the |RTI_SP_PRODUCT| in |COLLECTORSERVICE| depends 
on how you deployed |COLLECTORSERVICE| in your system.

Evaluation Deployment Using Docker Compose
""""""""""""""""""""""""""""""""""""""""""

If you installed the |OBSERVABILITY| in a single host using Docker Compose
(see :link_addon_products_observability:`Observability Framework for Evaluation <install.html#observability-heading-for-evaluation>`
in the *Connext Observability Framework User's Manual*), 
the |OBSERVABILITY| components are configured using a JSON file
the first time you run the ``rtiobservability`` script. Add the security
artifacts to your JSON configuration file, as shown in the following example:

.. code-block:: JSON
  :caption: Example JSON configuration file for |OBSERVABILITY| with |RTI_SP_PRODUCT|
  :emphasize-lines: 11-19


    {
        "hostname": "localhost",
        "securityConfig": {
            "basicAuthUsername": "...",
            "basicAuthPassword": "...",
            "httpsSecurity": {
                "caCertificate":     "...",
                "serverCertificate": "...",
                "serverKey":         "..."
            },
            "ddsSecurity": {
                "identityCaCertificate":    "path/to/identityCaCert.pem",
                "permissionsCaCertificate": "path/to/permissionsCaCert.pem",
                "identityCertificate":      "path/to/identityCert.pem",
                "identityKey":              "path/to/identityKey.pem",
                "signedPermissionsFile":    "path/to/signedPermissions.p7s",
                "signedGovernanceFile":     "path/to/signedGovernance.p7s",
                "pskSecretPassphrase":      "path/to/passphrase"
            }
        }
    }

See :link_addon_products_observability:`JSON Configuration File <install.html#json-configuration-file>`
in the *Connext Observability Framework User's Manual* for further 
information on the JSON configuration file syntax and usage.

If the JSON configuration contains the security artifacts, the |RTI_SP_PRODUCT| will be
enabled automatically in |COLLECTORSERVICE|.

.. note::
  With Docker Compose, |COLLECTORSERVICE| can only be configured with a 
  DDS receiver. Exporting to DDS is not supported.

Production Deployment
"""""""""""""""""""""

Collector Service Docker Image
++++++++++++++++++++++++++++++

If you are running |COLLECTORSERVICE| directly using the available Docker 
image (see :link_addon_products_observability:`Production Deployments <deployment.html#production-deployments>`
in the *Connext Observability Framework User's Manual*), 
the security artifacts must be bind-mounted when running the |COLLECTORSERVICE| Docker 
container, as shown in the following example:

.. code-block:: console
  :caption: Example bind-mounting of security artifacts for the |COLLECTORSERVICE| DDS receiver

    -v path/to/identityCaCert.pem:/rti/security/dds/identity_ca.pem
    -v path/to/permissionsCaCert.pem:/rti/security/dds/permissions_ca.pem
    -v path/to/identityCert.pem:/rti/security/dds/identity_certificate.pem
    -v path/to/identityKey.pem:/rti/security/dds/private_key.pem
    -v path/to/signedPermissions.p7s:/rti/security/dds/permissions.p7s
    -v path/to/signedGovernance.p7s:/rti/security/dds/governance.p7s
    -v /path/to/passphrase:/rti/security/dds/passphrase

.. code-block:: console
  :caption: Example bind-mounting of security artifacts for the |COLLECTORSERVICE| DDS exporter

    -v /path/to/identity_ca_output.pem:/rti/security/dds/identity_ca_output.pem
    -v /path/to/permissions_ca_output.pem:/rti/security/dds/permissions_ca_output.pem
    -v /path/to/identity_certificate_output.pem:/rti/security/dds/identity_certificate_output.pem
    -v /path/to/private_key_output.pem:/rti/security/dds/private_key_output.pem
    -v /path/to/governance_output.p7s:/rti/security/dds/governance_output.p7s
    -v /path/to/permissions_output.p7s:/rti/security/dds/permissions_output.p7s
    -v /path/to/passphrase_output:/rti/security/dds/passphrase_output

In addition, you need to set the ``CFG_NAME`` environment variable to one
of the provided Docker image's builtin  configurations that enable the
|RTI_SP_PRODUCT| in |COLLECTORSERVICE|. See
:link_addon_products_observability:`Collector Service Builtin Configurations <rest_api.html#tablecollectorservicebuiltinconfigurations>`
in the *Connext Observability Framework User's Manual* for  
details about the available builtin  configurations.

For additional information on how to use the |COLLECTORSERVICE| Docker 
image, refer to the `Collector Service article <https://hub.docker.com/repository/docker/rticom/collector-service/general>`_
in Docker Hub.

Collector Service Executable
++++++++++++++++++++++++++++

If you are running |COLLECTORSERVICE| as an executable 
(see :link_addon_products_observability:`Production Deployments <deployment.html#production-deployments>`
in the *Connext Observability Framework User's Manual*), the security artifacts must
be provided via the definition of environment variables, as shown in the following
example:

.. code-block:: console
  :caption: Example set environment variables for security artifacts for the |COLLECTORSERVICE| DDS receiver

    -DOBSERVABILITY_DDS_SECURITY_IDENTITY_CA="path/to/identity_ca.pem"
    -DOBSERVABILITY_DDS_SECURITY_PERMISSIONS_CA="path/to/permissions_ca.pem"
    -DOBSERVABILITY_DDS_SECURITY_IDENTITY_CERTIFICATE="path/to/identity_certificate.pem"
    -DOBSERVABILITY_DDS_SECURITY_PRIVATE_KEY="/path/to/private_key.pem"
    -DOBSERVABILITY_DDS_SECURITY_PERMISSIONS="/path/to/permissions.p7s"
    -DOBSERVABILITY_DDS_SECURITY_GOVERNANCE="/path/to/governance.p7s"
    -DOBSERVABILITY_DDS_SECURITY_PASSPHRASE="path/to/passphrase"

.. code-block:: console
  :caption: Example set environment variables for security artifacts for the |COLLECTORSERVICE| DDS exporter

    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_IDENTITY_CA="path/to/identity_ca_output.pem"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PERMISSIONS_CA="path/to/permissions_ca_output.pem"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_IDENTITY_CERTIFICATE="path/to/identity_certificate_output.pem"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PRIVATE_KEY="/path/to/private_key_output.pem"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PERMISSIONS="/path/to/permissions_output.p7s"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_GOVERNANCE="/path/to/governance_output.p7s"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PASSPHRASE="path/to/passphrase_output"

In addition, you need to set the ``-cfgName`` environment variable to one
of the provided Docker image's builtin  configurations that enable the
|RTI_SP_PRODUCT| in |COLLECTORSERVICE|. See
:link_addon_products_observability:`Collector Service Builtin Configurations <rest_api.html#tablecollectorservicebuiltinconfigurations>`
in the *Connext Observability Framework User's Manual* for details about the available
builtin  configurations.

For additional information on how to use the |COLLECTORSERVICE|, refer to
:link_addon_products_observability:`Collector Service <rest_api.html#collector-service>`
chapter in the *Connext Observability Framework User's Manual*.

Enabling |RTI_SP_PRODUCT_HEADING| in |MONITORINGLIBRARY2_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To enable the |RTI_SP_PRODUCT| in |MONITORINGLIBRARY2|, you need to define a QoS profile with
the properties related to security artifacts (|RTI_SP_PRODUCT| enablement, certificates,
|GovernanceDoc|, and |PermissionsDoc|). Then, you need to set
that profile name in
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name``
(see :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
for further information on this field). For example:

.. literalinclude:: ../snippets/observability_monitoring_qos_profile_with_security.xml
  :language: xml
  :emphasize-lines: 60-62
  :caption: Example QoS configuration for enabling |RTI_SP_PRODUCT| in |MONITORINGLIBRARY2|

.. note::
  You can see an example of how to enable the |RTI_SP_PRODUCT| in |MONITORINGLIBRARY2| in the
  shipped Temperature example (``<path to examples>/observability/c++``). This
  example is described in detail in the
  :link_addon_products_observability:`Tutorial <gettingstarted.html>` in the
  *Connext Observability Framework User's Manual*.

Securing Telemetry Data with |LIGHT_SP_BUILTIN_HEADING|
-------------------------------------------------------

Evaluation Deployment Using Docker Compose with  |LIGHT_SP_BUILTIN_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. important::
  
    |LIGHT_SP_BUILTIN| **is not supported** for |OBSERVABILITY| evaluation
    deployments.

Production Deployment with  |LIGHT_SP_BUILTIN_HEADING|
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Collector Service Docker Image with  |LIGHT_SP_BUILTIN_HEADING|
""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

If you are running |COLLECTORSERVICE| directly using the available Docker 
image (see :link_addon_products_observability:`Production Deployments <deployment.html#production-deployments>`
in the *Connext Observability Framework User's Manual*), 
the security artifacts must be bind-mounted when running the |COLLECTORSERVICE| Docker 
container, as shown in the following example:

.. code-block:: console
  :caption: Example bind-mounting of security artifacts for the |COLLECTORSERVICE| DDS receiver

    -v /path/to/passphrase:/rti/security/dds/passphrase

.. code-block:: console
  :caption: Example bind-mounting of security artifacts for the |COLLECTORSERVICE| DDS exporter

    -v /path/to/passphrase_output:/rti/security/dds/passphrase_output

In addition, you need to set the ``CFG_NAME`` environment variable to one
of the provided Docker image's builtin  configurations that enable the
|RTI_SP_PRODUCT| in |COLLECTORSERVICE|. See
:link_addon_products_observability:`Collector Service Builtin Configurations <rest_api.html#tablecollectorservicebuiltinconfigurations>`
in the *Connext Observability Framework User's Manual* for  
details about the available builtin  configurations.

The following environment variables may optionally be changed to configure the
protection kind and cipher for the |LIGHT_SP_BUILTIN|:

.. code-block:: console
  :caption: Example environment variables to configure the protection kind and cipher for the |LIGHT_SP_BUILTIN| in |COLLECTORSERVICE| using Docker image

    -e OBSERVABILITY_DDS_SECURITY_PSK_PROTECTION_KIND="<protection_kind>"
    -e OBSERVABILITY_DDS_SECURITY_PSK_CIPHER="<cipher>"
    -e OBSERVABILITY_OUTPUT_DDS_SECURITY_PSK_PROTECTION_KIND="<protection_kind>"
    -e OBSERVABILITY_OUTPUT_DDS_SECURITY_PSK_CIPHER="<cipher>"

For additional information on values for protection kind and cipher, see
:ref:`p2_core/cryptography:rtps_psk_protection_kind (domain_rule)` and
:ref:`section-supported-cryptographic-algorithms`
respectively.

For additional information on how to use the |COLLECTORSERVICE| Docker 
image, refer to the `Collector Service article <https://hub.docker.com/repository/docker/rticom/collector-service/general>`_
in Docker Hub.

Collector Service Executable with  |LIGHT_SP_BUILTIN_HEADING|
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

If you are running |COLLECTORSERVICE| as an executable, the security artifacts must
be provided via the definition of environment variables, as shown in the following
example:

.. code-block:: console
  :caption: Example set environment variables for security artifacts for the |COLLECTORSERVICE| DDS receiver

    -DOBSERVABILITY_DDS_SECURITY_PASSPHRASE="path/to/passphrase"

.. code-block:: console
  :caption: Example set environment variables for security artifacts for the |COLLECTORSERVICE| DDS exporter

    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PASSPHRASE="path/to/passphrase_output"

In addition, you need to set the ``-cfgName`` environment variable to one
of the provided Docker image's builtin  configurations that enable the
|LIGHT_SP_BUILTIN| in |COLLECTORSERVICE|. See
:link_addon_products_observability:`Collector Service Builtin Configurations <rest_api.html#tablecollectorservicebuiltinconfigurations>`
in the *Connext Observability Framework User's Manual* for details about the available
builtin  configurations.

The following environment variables may optionally be changed to configure the
protection kind and cipher for the |LIGHT_SP_BUILTIN|:

.. code-block:: console
  :caption: Example environment variables to configure the protection kind and cipher for the |LIGHT_SP_BUILTIN| in |COLLECTORSERVICE| using executable

    -DOBSERVABILITY_DDS_SECURITY_PSK_PROTECTION_KIND="<protection_kind>"
    -DOBSERVABILITY_DDS_SECURITY_PSK_CIPHER="<cipher>"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PSK_PROTECTION_KIND="<protection_kind>"
    -DOBSERVABILITY_OUTPUT_DDS_SECURITY_PSK_CIPHER="<cipher>"

For additional information on values for protection kind and cipher, see
:ref:`p2_core/cryptography:rtps_psk_protection_kind (domain_rule)` and
:ref:`section-supported-cryptographic-algorithms`
respectively.

For additional information on how to use the |COLLECTORSERVICE|, refer to
:link_addon_products_observability:`Collector Service <rest_api.html#collector-service>`
chapter in the *Connext Observability Framework User's Manual*.

Enabling |LIGHT_SP_BUILTIN_HEADING| in |MONITORINGLIBRARY2_HEADING|
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

To enable the |RTI_SP_PRODUCT| in |MONITORINGLIBRARY2|, you need to define a QoS profile with
the properties related to security artifacts (|LIGHT_SP_BUILTIN| configuration and passphrase). Then,
you need to set that profile name in
``participant_factory_qos.monitoring.distribution_settings.dedicated_participant.participant_qos_profile_name``
(see :link_connext_dds_pro_um:`MONITORING QosPolicy (DDS Extension) <#users_manual/MONITORING_QosPolicy.htm>`
for further information on this field). For example:

.. literalinclude:: ../snippets/observability_monitoring_qos_profile_with_lightweight_security.xml
  :language: xml
  :emphasize-lines: 48-50
  :caption: Example QoS configuration for enabling |LIGHT_SP_BUILTIN| in |MONITORINGLIBRARY2|

.. note::
  You can see an example of how to enable the |LIGHT_SP_BUILTIN| in |MONITORINGLIBRARY2| in the
  shipped Temperature example (``<path to examples>/observability/c++``). This
  example is described in detail in the
  :link_addon_products_observability:`Tutorial <gettingstarted.html>` in the
  *Connext Observability Framework User's Manual*.
