You may notice some articles displaying content inconsistently. Pardon our dust as we update our site.
cross icon
In this article
dropdown icon
Preface
    New and changed information
    dropdown icon
    Get Started with Hybrid Data Security
      Hybrid Data Security Overview
        dropdown icon
        Security Realm Architecture
          Realms of Separation (without Hybrid Data Security)
        Collaborating with Other Organizations
          Expectations for Deploying Hybrid Data Security
            High-level Setup Process
              dropdown icon
              Hybrid Data Security Deployment Model
                Hybrid Data Security Deployment Model
              Hybrid Data Security Trial Mode
                dropdown icon
                Standby Data Center for Disaster Recovery
                  Setup Standby Data Center for Disaster Recovery
                Proxy Support
                dropdown icon
                Prepare Your Environment
                  dropdown icon
                  Requirements for Hybrid Data Security
                    Cisco Webex License Requirements
                    Docker Desktop Requirements
                    X.509 Certificate Requirements
                    Virtual Host Requirements
                    Database server requirements
                    External connectivity requirements
                    Proxy Server Requirements
                  Complete the Prerequisites for Hybrid Data Security
                  dropdown icon
                  Set up a Hybrid Data Security Cluster
                    Hybrid Data Security Deployment Task Flow
                      Download Installation Files
                        Create a Configuration ISO for the HDS Hosts
                          Install the HDS Host OVA
                            Set up the Hybrid Data Security VM
                              Upload and Mount the HDS Configuration ISO
                                Configure the HDS Node for Proxy Integration
                                  Register the First Node in the Cluster
                                    Create and Register More Nodes
                                    dropdown icon
                                    Run a Trial and Move to Production
                                      Trial to Production Task Flow
                                        Activate Trial
                                          Test Your Hybrid Data Security Deployment
                                            Monitor Hybrid Data Security Health
                                              Add or Remove Users from Your Trial
                                                Move from Trial to Production
                                                  End Your Trial Without Moving to Production
                                                  dropdown icon
                                                  Manage your HDS Deployment
                                                    Manage HDS Deployment
                                                      Set Cluster Upgrade Schedule
                                                        Change the Node Configuration
                                                          Turn off Blocked External DNS Resolution Mode
                                                            Remove a Node
                                                              Disaster Recovery using Standby Data Center
                                                                (Optional) Unmount ISO After HDS Configuration
                                                                dropdown icon
                                                                Troubleshoot Hybrid Data Security
                                                                  View Alerts and Troubleshoot
                                                                    dropdown icon
                                                                    Alerts
                                                                      Common Issues and the Steps to Resolve Them
                                                                    Troubleshoot Hybrid Data Security
                                                                    dropdown icon
                                                                    Other Notes
                                                                      Known Issues for Hybrid Data Security
                                                                        Use OpenSSL to Generate a PKCS12 File
                                                                          Traffic between the HDS Nodes and the Cloud
                                                                            dropdown icon
                                                                            Configure Squid Proxies for Hybrid Data Security
                                                                              Websocket Cannot Connect Through Squid Proxy
                                                                          In this article
                                                                          cross icon
                                                                          dropdown icon
                                                                          Preface
                                                                            New and changed information
                                                                            dropdown icon
                                                                            Get Started with Hybrid Data Security
                                                                              Hybrid Data Security Overview
                                                                                dropdown icon
                                                                                Security Realm Architecture
                                                                                  Realms of Separation (without Hybrid Data Security)
                                                                                Collaborating with Other Organizations
                                                                                  Expectations for Deploying Hybrid Data Security
                                                                                    High-level Setup Process
                                                                                      dropdown icon
                                                                                      Hybrid Data Security Deployment Model
                                                                                        Hybrid Data Security Deployment Model
                                                                                      Hybrid Data Security Trial Mode
                                                                                        dropdown icon
                                                                                        Standby Data Center for Disaster Recovery
                                                                                          Setup Standby Data Center for Disaster Recovery
                                                                                        Proxy Support
                                                                                        dropdown icon
                                                                                        Prepare Your Environment
                                                                                          dropdown icon
                                                                                          Requirements for Hybrid Data Security
                                                                                            Cisco Webex License Requirements
                                                                                            Docker Desktop Requirements
                                                                                            X.509 Certificate Requirements
                                                                                            Virtual Host Requirements
                                                                                            Database server requirements
                                                                                            External connectivity requirements
                                                                                            Proxy Server Requirements
                                                                                          Complete the Prerequisites for Hybrid Data Security
                                                                                          dropdown icon
                                                                                          Set up a Hybrid Data Security Cluster
                                                                                            Hybrid Data Security Deployment Task Flow
                                                                                              Download Installation Files
                                                                                                Create a Configuration ISO for the HDS Hosts
                                                                                                  Install the HDS Host OVA
                                                                                                    Set up the Hybrid Data Security VM
                                                                                                      Upload and Mount the HDS Configuration ISO
                                                                                                        Configure the HDS Node for Proxy Integration
                                                                                                          Register the First Node in the Cluster
                                                                                                            Create and Register More Nodes
                                                                                                            dropdown icon
                                                                                                            Run a Trial and Move to Production
                                                                                                              Trial to Production Task Flow
                                                                                                                Activate Trial
                                                                                                                  Test Your Hybrid Data Security Deployment
                                                                                                                    Monitor Hybrid Data Security Health
                                                                                                                      Add or Remove Users from Your Trial
                                                                                                                        Move from Trial to Production
                                                                                                                          End Your Trial Without Moving to Production
                                                                                                                          dropdown icon
                                                                                                                          Manage your HDS Deployment
                                                                                                                            Manage HDS Deployment
                                                                                                                              Set Cluster Upgrade Schedule
                                                                                                                                Change the Node Configuration
                                                                                                                                  Turn off Blocked External DNS Resolution Mode
                                                                                                                                    Remove a Node
                                                                                                                                      Disaster Recovery using Standby Data Center
                                                                                                                                        (Optional) Unmount ISO After HDS Configuration
                                                                                                                                        dropdown icon
                                                                                                                                        Troubleshoot Hybrid Data Security
                                                                                                                                          View Alerts and Troubleshoot
                                                                                                                                            dropdown icon
                                                                                                                                            Alerts
                                                                                                                                              Common Issues and the Steps to Resolve Them
                                                                                                                                            Troubleshoot Hybrid Data Security
                                                                                                                                            dropdown icon
                                                                                                                                            Other Notes
                                                                                                                                              Known Issues for Hybrid Data Security
                                                                                                                                                Use OpenSSL to Generate a PKCS12 File
                                                                                                                                                  Traffic between the HDS Nodes and the Cloud
                                                                                                                                                    dropdown icon
                                                                                                                                                    Configure Squid Proxies for Hybrid Data Security
                                                                                                                                                      Websocket Cannot Connect Through Squid Proxy

                                                                                                                                                  Deployment guide for Webex Hybrid Data Security

                                                                                                                                                  list-menuIn this article

                                                                                                                                                  Preface

                                                                                                                                                  New and changed information

                                                                                                                                                  Date

                                                                                                                                                  Changes Made

                                                                                                                                                  October 20, 2023

                                                                                                                                                  August 07, 2023

                                                                                                                                                  May 23, 2023

                                                                                                                                                  December 06, 2022

                                                                                                                                                  November 23, 2022

                                                                                                                                                  October 13, 2021

                                                                                                                                                  Docker Desktop needs to run a setup program before you can install HDS nodes. See Docker Desktop Requirements.

                                                                                                                                                  June 24, 2021

                                                                                                                                                  Noted that you can reuse the private key file and CSR to request another certificate. See Use OpenSSL to Generate a PKCS12 File for details.

                                                                                                                                                  April 30, 2021

                                                                                                                                                  Changed the VM requirement for local hard disk space to 30 GB. See Virtual Host Requirements for details.

                                                                                                                                                  February 24, 2021

                                                                                                                                                  HDS Setup Tool can now run behind a proxy. See Create a Configuration ISO for the HDS Hosts for details.

                                                                                                                                                  February 2, 2021

                                                                                                                                                  HDS can now run without a mounted ISO file. See (Optional) Unmount ISO After HDS Configuration for details.

                                                                                                                                                  January 11, 2021

                                                                                                                                                  Added info on HDS Setup tool and proxies to Create a Configuration ISO for the HDS Hosts.

                                                                                                                                                  October 13, 2020

                                                                                                                                                  Updated Download Installation Files.

                                                                                                                                                  October 8, 2020

                                                                                                                                                  Updated Create a Configuration ISO for the HDS Hosts and Change the Node Configuration with commands for FedRAMP environments.

                                                                                                                                                  August 14, 2020

                                                                                                                                                  Updated Create a Configuration ISO for the HDS Hosts and Change the Node Configuration with changes to the sign-in process.

                                                                                                                                                  August 5, 2020

                                                                                                                                                  Updated Test Your Hybrid Data Security Deployment for changes in log messages.

                                                                                                                                                  Updated Virtual Host Requirements to remove maximum number of hosts.

                                                                                                                                                  June 16, 2020

                                                                                                                                                  Updated Remove a Node for changes in the Control Hub UI.

                                                                                                                                                  June 4, 2020

                                                                                                                                                  Updated Create a Configuration ISO for the HDS Hosts for changes in the Advanced Settings that you might set.

                                                                                                                                                  May 29, 2020

                                                                                                                                                  Updated Create a Configuration ISO for the HDS Hosts to show you can also use TLS with SQL Server databases, UI changes, and other clarifications.

                                                                                                                                                  May 5, 2020

                                                                                                                                                  Updated Virtual Host Requirements to show new requirement of ESXi 6.5.

                                                                                                                                                  April 21, 2020

                                                                                                                                                  Updated External connectivity requirements with new Americas CI hosts.

                                                                                                                                                  April 1, 2020

                                                                                                                                                  Updated External connectivity requirements with information on regional CI hosts.

                                                                                                                                                  February 20, 2020Updated Create a Configuration ISO for the HDS Hosts with information on new optional Advanced Settings screen in the HDS Setup Tool.
                                                                                                                                                  February 4, 2020Updated Proxy Server Requirements.
                                                                                                                                                  December 16, 2019Clarified the requirement for Blocked External DNS Resolution Mode to work in Proxy Server Requirements.
                                                                                                                                                  November 19, 2019

                                                                                                                                                  Added information about Blocked External DNS Resolution Mode in the following sections:

                                                                                                                                                  November 8, 2019

                                                                                                                                                  You can now configure network settings for a node while deploying the OVA rather than afterwards.

                                                                                                                                                  Updated the following sections accordingly:

                                                                                                                                                  The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions.

                                                                                                                                                  September 6, 2019

                                                                                                                                                  Added SQL Server Standard to Database server requirements.

                                                                                                                                                  August 29, 2019Added Configure Squid Proxies for Hybrid Data Security appendix with guidance on configuring Squid proxies to ignore websocket traffic for proper operation.
                                                                                                                                                  August 20, 2019

                                                                                                                                                  Added and updated sections to cover proxy support for Hybrid Data Security node communications to the Webex cloud.

                                                                                                                                                  To access just the proxy support content for an existing deployment, see the Proxy Support for Hybrid Data Security and Webex Video Mesh help article.

                                                                                                                                                  June 13, 2019Updated Trial to Production Task Flow with a reminder to synchronize the HdsTrialGroup group object before starting a trial if your organization uses directory synchronization.
                                                                                                                                                  March 6, 2019
                                                                                                                                                  February 28, 2019
                                                                                                                                                  • Corrected the amount of local hard disk space per server that you should set aside when preparing the virtual hosts that become the Hybrid Data Security nodes, from 50-GB to 20-GB, to reflect the size of disk that the OVA creates.

                                                                                                                                                  February 26, 2019
                                                                                                                                                  • Hybrid Data Security nodes now support encrypted connections with PostgreSQL database servers, and encrypted logging connections to a TLS-capable syslog server. Updated Create a Configuration ISO for the HDS Hosts with instructions.

                                                                                                                                                  • Removed destination URLs from the "Internet Connectivity Requirements for Hybrid Data Security Node VMs" table. The table now refers to the list maintained in the "Additional URLs for Webex Teams Hybrid Services" table of Network Requirements for Webex Teams Services.

                                                                                                                                                  January 24, 2019

                                                                                                                                                  • Hybrid Data Security now supports Microsoft SQL Server as a database. SQL Server Always On (Always On Failover Clusters and Always on Availability Groups) is supported by the JDBC drivers that are used in Hybrid Data Security. Added content related to deploying with SQL Server.

                                                                                                                                                    Microsoft SQL Server support is intended for new deployments of Hybrid Data Security only. We do not currently support migration of data from PostgreSQL to Microsoft SQL Server in an existing deployment.

                                                                                                                                                  November 5, 2018
                                                                                                                                                  October 19, 2018

                                                                                                                                                  July 31, 2018

                                                                                                                                                  May 21, 2018

                                                                                                                                                  Changed terminology to reflect the rebranding of Cisco Spark:

                                                                                                                                                  • Cisco Spark Hybrid Data Security is now Hybrid Data Security.

                                                                                                                                                  • The Cisco Spark app is now the Webex App app.

                                                                                                                                                  • The Cisco Collaboraton Cloud is now the Webex cloud.

                                                                                                                                                  April 11, 2018
                                                                                                                                                  February 22, 2018
                                                                                                                                                  February 15, 2018
                                                                                                                                                  • In the X.509 Certificate Requirements table, specified that the certificate cannot be a wildcard certificate, and that the KMS uses the CN domain, not any domain that's defined in the x.509v3 SAN fields.

                                                                                                                                                  January 18, 2018

                                                                                                                                                  November 2, 2017

                                                                                                                                                  • Clarified directory synchronization of the HdsTrialGroup.

                                                                                                                                                  • Fixed instructions for uploading the ISO configuration file for mounting to the VM nodes.

                                                                                                                                                  August 18, 2017

                                                                                                                                                  First published

                                                                                                                                                  Get Started with Hybrid Data Security

                                                                                                                                                  Hybrid Data Security Overview

                                                                                                                                                  From day one, data security has been the primary focus in designing Webex App. The cornerstone of this security is end-to-end content encryption, enabled by Webex App clients interacting with the Key Management Service (KMS). The KMS is responsible for creating and managing the cryptographic keys that clients use to dynamically encrypt and decrypt messages and files.

                                                                                                                                                  By default, all Webex App customers get end-to-end encryption with dynamic keys stored in the cloud KMS, in Cisco's security realm. Hybrid Data Security moves the KMS and other security-related functions to your enterprise data center, so nobody but you holds the keys to your encrypted content.

                                                                                                                                                  Security Realm Architecture

                                                                                                                                                  The Webex cloud architecture separates different types of service into separate realms, or trust domains, as depicted below.

                                                                                                                                                  Realms of Separation (without Hybrid Data Security)

                                                                                                                                                  To further understand Hybrid Data Security, let's first look at this pure cloud case, where Cisco is providing all functions in its cloud realms. The identity service, the only place where users can be directly correlated with their personal information such as email address, is logically and physically separate from the security realm in data center B. Both are in turn separate from the realm where encrypted content is ultimately stored, in data center C.

                                                                                                                                                  In this diagram, the client is the Webex App running on a user's laptop, and has authenticated with the identity service. When the user composes a message to send to a space, the following steps take place:

                                                                                                                                                  1. The client establishes a secure connection with the key management service (KMS), then requests a key to encrypt the message. The secure connection uses ECDH, and the KMS encrypts the key using an AES-256 master key.

                                                                                                                                                  2. The message is encrypted before it leaves the client. The client sends it to the indexing service, which creates encrypted search indexes to aid in future searches for the content.

                                                                                                                                                  3. The encrypted message is sent to the compliance service for compliance checks.

                                                                                                                                                  4. The encrypted message is stored in the storage realm.

                                                                                                                                                  When you deploy Hybrid Data Security, you move the security realm functions (KMS, indexing, and compliance) to your on-premises data center. The other cloud services that make up Webex (including identity and content storage) remain in Cisco’s realms.

                                                                                                                                                  Collaborating with Other Organizations

                                                                                                                                                  Users in your organization may regularly use Webex App to collaborate with external participants in other organizations. When one of your users requests a key for a space that is owned by your organization (because it was created by one of your users) your KMS sends the key to the client over an ECDH secured channel. However, when another organization owns the key for the space, your KMS routes the request out to the Webex cloud through a separate ECDH channel to get the key from the appropriate KMS, and then returns the key to your user on the original channel.

                                                                                                                                                  The KMS service running on Org A validates the connections to KMSs in other organizations using x.509 PKI certificates. See Prepare Your Environment for details on generating an x.509 certificate to use with your Hybrid Data Security deployment.

                                                                                                                                                  Expectations for Deploying Hybrid Data Security

                                                                                                                                                  High-level Setup Process

                                                                                                                                                  This document covers the setup and management of a Hybrid Data Security deployment:

                                                                                                                                                  • Set up Hybrid Data Security—This includes preparing required infrastructure and installing Hybrid Data Security software, testing your deployment with a subset of users in trial mode, and, once your testing is complete, moving to production. This converts the entire organization to use your Hybrid Data Security cluster for security functions.

                                                                                                                                                    The setup, trial, and production phases are covered in detail in the next three chapters.

                                                                                                                                                  • Maintain your Hybrid Data Security deployment—The Webex cloud automatically provides ongoing upgrades. Your IT department can provide tier one support for this deployment, and engage Cisco support as needed. You can use on-screen notifications and set up email-based alerts in Control Hub.

                                                                                                                                                  • Understand common alerts, troubleshooting steps, and known issues—If you run into trouble deploying or using Hybrid Data Security, the last chapter of this guide and the Known Issues appendix may help you determine and fix the issue.

                                                                                                                                                  Hybrid Data Security Deployment Model

                                                                                                                                                  Within your enterprise data center, you deploy Hybrid Data Security as a single cluster of nodes on separate virtual hosts. The nodes communicate with the Webex cloud through secure websockets and secure HTTP.

                                                                                                                                                  During the installation process, we provide you with the OVA file to set up the virtual appliance on the VMs that you provide. You use the HDS Setup Tool to create a custom cluster configuration ISO file that you mount on each node. The Hybrid Data Security cluster uses your provided Syslogd server and PostgreSQL or Microsoft SQL Server database. (You configure the Syslogd and database connection details in the HDS Setup Tool.)

                                                                                                                                                  Hybrid Data Security Deployment Model

                                                                                                                                                  The minimum number of nodes you can have in a cluster is two. We recommend at least three per cluster. Having multiple nodes ensures that service is not interrupted during a software upgrade or other maintenance activity on a node. (The Webex cloud only upgrades one node at a time.)

                                                                                                                                                  All nodes in a cluster access the same key datastore, and log activity to the same syslog server. The nodes themselves are stateless, and handle key requests in round-robin fashion, as directed by the cloud.

                                                                                                                                                  Nodes become active when you register them in Control Hub. To take an individual node out of service, you can deregister it, and later reregister it if needed.

                                                                                                                                                  We support only a single cluster per organization.

                                                                                                                                                  Hybrid Data Security Trial Mode

                                                                                                                                                  After setting up a Hybrid Data Security deployment, you first try it with a set of pilot users. During the trial period, these users use your on-premises Hybrid Data Security domain for encryption keys and other security realm services. Your other users continue to use the cloud security realm.

                                                                                                                                                  If you decide not to continue with the deployment during the trial and deactivate the service, the pilot users and any users they have interacted with by creating new spaces during the trial period will lose access to the messages and content. They will see “This message cannot be decrypted” in the Webex App.

                                                                                                                                                  If you are satisfied that your deployment is working well for the trial users and you are ready to extend Hybrid Data Security to all of your users, you move the deployment to production. Pilot users continue to have access to the keys that were in use during the trial. However, you cannot move back and forth between production mode and the original trial. If you must deactivate the service, such as to perform disaster recovery, when you reactivate you must start a new trial and set up the set of pilot users for the new trial before moving back to production mode. Whether users retain access to data at this point depends on whether you have successfully maintained backups of the key data store and the ISO configuration file for the Hybrid Data Security nodes in your cluster.

                                                                                                                                                  Standby Data Center for Disaster Recovery

                                                                                                                                                  During deployment, you set up a secure standby data center. In the event of a data center disaster, you can manually fail your deployment over to the standby data center.

                                                                                                                                                  Before failover, Data Center A has active HDS nodes and the primary PostgreSQL or Microsoft SQL Server database, while B has a copy of the ISO file with additional configurations, VMs that are registered to the organization, and a standby database. After failover, Data Center B has active HDS nodes and the primary database, while A has unregistered VMs and a copy of the ISO file, and the database is in standby mode.
                                                                                                                                                  Manual Failover to Standby Data Center

                                                                                                                                                  The databases of the active and standby data centers are in sync with each other which will minimize the time taken to perform the failover. The ISO file of the standby data center is updated with additional configurations which ensure that the nodes are registered to the organization, but will not handle traffic. Hence, the nodes of the standby data center always remain up-to-date with the latest version of HDS software.

                                                                                                                                                  The active Hybrid Data Security nodes must always be in the same data center as the active database server.

                                                                                                                                                  Setup Standby Data Center for Disaster Recovery

                                                                                                                                                  Follow the steps below to configure the ISO file of the standby data center:

                                                                                                                                                  1

                                                                                                                                                  Start the HDS Setup tool and follow the steps mentioned in Create a Configuration ISO for the HDS Hosts.

                                                                                                                                                  The ISO file must be a copy of the original ISO file of the primary data center onto which the following configuration updates are to be made.

                                                                                                                                                  2

                                                                                                                                                  After configuring the Syslogd server, click on Advanced Settings

                                                                                                                                                  3

                                                                                                                                                  On the Advanced Settings page, add the configuration below to place the node in passive mode. In this mode the node will be registered to the organization and connected to cloud, but will not handle any traffic.

                                                                                                                                                  
                                                                                                                                                  passiveMode: 'true'
                                                                                                                                                  

                                                                                                                                                  4

                                                                                                                                                  Complete the configuration process and save the ISO file in a location that's easy to find.

                                                                                                                                                  5

                                                                                                                                                  Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes.

                                                                                                                                                  6

                                                                                                                                                  In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings..

                                                                                                                                                  7

                                                                                                                                                  Click Edit Settings >CD/DVD Drive 1 and select Datastore ISO File.

                                                                                                                                                  Make sure Connected and Connect at power on are checked so that updated configuration changes can take effect after starting the nodes.

                                                                                                                                                  8

                                                                                                                                                  Power on the HDS node and make sure there are no alarms for at least 15 minutes.

                                                                                                                                                  9

                                                                                                                                                  Repeat the process for every node in the standby data center.

                                                                                                                                                  Check the syslogs to verify that the nodes are in passive mode. You should be able to view the message “KMS configured in passive mode” in the syslogs.

                                                                                                                                                  After configuring passiveMode in the ISO file and saving it, you can create another copy of the ISO file without the passiveMode configuration and save it in a secure location. This copy of the ISO file without passiveMode configured can help in a quick failover process during disaster recovery. See Disaster Recovery using Standby Data Center for the detailed failover procedure.

                                                                                                                                                  Proxy Support

                                                                                                                                                  Hybrid Data Security supports explicit, transparent inspecting, and non-inspecting proxies. You can tie these proxies to your deployment so that you can secure and monitor traffic from the enterprise out to the cloud. You can use a platform admin interface on the nodes for certificate management and to check the overall connectivity status after you set up the proxy on the nodes.

                                                                                                                                                  The Hybrid Data Security nodes support the following proxy options:

                                                                                                                                                  • No proxy—The default if you do not use the HDS node setup Trust Store & Proxy configuration to integrate a proxy. No certificate update is required.

                                                                                                                                                  • Transparent non-inspecting proxy—The nodes are not configured to use a specific proxy server address and should not require any changes to work with a non-inspecting proxy. No certificate update is required.

                                                                                                                                                  • Transparent tunneling or inspecting proxy—The nodes are not configured to use a specific proxy server address. No HTTP or HTTPS configuration changes are necessary on the nodes. However, the nodes need a root certificate so that they trust the proxy. Inspecting proxies are typically used by IT to enforce policies on which websites can be visited and which types of content are not permitted. This type of proxy decrypts all your traffic (even HTTPS).

                                                                                                                                                  • Explicit proxy—With explicit proxy, you tell the HDS nodes which proxy server and authentication scheme to use. To configure an explicit proxy, you must enter the following information on each node:

                                                                                                                                                    1. Proxy IP/FQDN—Address that can be used to reach the proxy machine.

                                                                                                                                                    2. Proxy Port—A port number that the proxy uses to listen for proxied traffic.

                                                                                                                                                    3. Proxy Protocol—Depending on what your proxy server supports, choose between the following protocols:

                                                                                                                                                      • HTTP—Views and controls all requests that the client sends.

                                                                                                                                                      • HTTPS—Provides a channel to the server. The client receives and validates the server's certificate.

                                                                                                                                                    4. Authentication Type—Choose from among the following authentication types:

                                                                                                                                                      • None—No further authentication is required.

                                                                                                                                                        Available if you select either HTTP or HTTPS as the proxy protocol.

                                                                                                                                                      • Basic—Used for an HTTP User Agent to provide a user name and password when making a request. Uses Base64 encoding.

                                                                                                                                                        Available if you select either HTTP or HTTPS as the proxy protocol.

                                                                                                                                                        Requires you to enter the user name and password on each node.

                                                                                                                                                      • Digest—Used to confirm the account before sending sensitive information. Applies a hash function on the user name and password before sending over the network.

                                                                                                                                                        Available only if you select HTTPS as the proxy protocol.

                                                                                                                                                        Requires you to enter the user name and password on each node.

                                                                                                                                                  Example of Hybrid Data Security Nodes and Proxy

                                                                                                                                                  This diagram shows an example connection between the Hybrid Data Security, network and a proxy. For the transparent inspecting and HTTPS explicit inspecting proxy options, the same root certificate must be installed on the proxy and on the Hybrid Data Security nodes.

                                                                                                                                                  Blocked External DNS Resolution Mode (Explicit Proxy Configurations)

                                                                                                                                                  When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. In deployments with explicit proxy configurations that do not allow external DNS resolution for internal clients, if the node can't query the DNS servers, it automatically goes into Blocked External DNS Resolution mode. In this mode, node registration and other proxy connectivity tests can proceed.

                                                                                                                                                  Prepare Your Environment

                                                                                                                                                  Requirements for Hybrid Data Security

                                                                                                                                                  Docker Desktop Requirements

                                                                                                                                                  Before you install your HDS nodes, you need Docker Desktop to run a setup program. Docker recently updated their licensing model. Your organization might require a paid subscription for Docker Desktop. For details, see the Docker blog post, " Docker is Updating and Extending Our Product Subscriptions".

                                                                                                                                                  X.509 Certificate Requirements

                                                                                                                                                  The certificate chain must meet the following requirements:

                                                                                                                                                  Table 1. X.509 Certificate Requirements for Hybrid Data Security Deployment

                                                                                                                                                  Requirement

                                                                                                                                                  Details

                                                                                                                                                  • Signed by a trusted Certificate Authority (CA)

                                                                                                                                                  By default, we trust the CAs in the Mozilla list (with the exception of WoSign and StartCom) at https://wiki.mozilla.org/CA:IncludedCAs.

                                                                                                                                                  • Bears a Common Name (CN) domain name that identifies your Hybrid Data Security deployment

                                                                                                                                                  • Is not a wildcard certificate

                                                                                                                                                  The CN does not need to be reachable or a live host. We recommend that you use a name which reflects your organization, for example, hds.company.com.

                                                                                                                                                  The CN must not contain a * (wildcard).

                                                                                                                                                  The CN is used to verify the Hybrid Data Security nodes to Webex App clients. All of the Hybrid Data Security nodes in your cluster use the same certificate. Your KMS identifies itself using the CN domain, not any domain that is defined in the x.509v3 SAN fields.

                                                                                                                                                  Once you have registered a node with this certificate, we do not support changing the CN domain name. Choose a domain that can apply to both the trial and production deployments.

                                                                                                                                                  • Non-SHA1 signature

                                                                                                                                                  The KMS software does not support SHA1 signatures for validating connections to other organizations' KMSs.

                                                                                                                                                  • Formatted as a password-protected PKCS #12 file

                                                                                                                                                  • Use the friendly name of kms-private-key to tag the certificate, private key, and any intermediate certificates to upload.

                                                                                                                                                  You can use a converter such as OpenSSL to change your certificate's format.

                                                                                                                                                  You will need to enter the password when you run the HDS Setup Tool.

                                                                                                                                                  The KMS software does not enforce key usage or extended key usage constraints. Some certificate authorities require that extended key usage constraints be applied to each certificate, such as server authentication. It is okay to use the server authentication or other settings.

                                                                                                                                                  Virtual Host Requirements

                                                                                                                                                  The virtual hosts that you will set up as Hybrid Data Security nodes in your cluster have the following requirements:

                                                                                                                                                  • At least two separate hosts (3 recommended) colocated in the same secure data center

                                                                                                                                                  • VMware ESXi 6.5 (or later) installed and running.

                                                                                                                                                    You must upgrade if you have an earlier version of ESXi.

                                                                                                                                                  • Minimum 4 vCPUs, 8-GB main memory, 30-GB local hard disk space per server

                                                                                                                                                  Database server requirements

                                                                                                                                                  Create a new database for key storage. Don’t use the default database. The HDS applications, when installed, create the database schema.

                                                                                                                                                  There are two options for database server. The requirements for each are as follows:

                                                                                                                                                  Table 2. Database server requirements by type of database

                                                                                                                                                  PostgreSQL

                                                                                                                                                  Microsoft SQL Server

                                                                                                                                                  • PostgreSQL 14, 15, or 16, installed and running.

                                                                                                                                                  • SQL Server 2016, 2017, or 2019 (Enterprise or Standard) installed.

                                                                                                                                                    SQL Server 2016 requires Service Pack 2 and Cumulative Update 2 or later.

                                                                                                                                                  Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage)

                                                                                                                                                  Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage)

                                                                                                                                                  The HDS software currently installs the following driver versions for communication with the database server:

                                                                                                                                                  PostgreSQL

                                                                                                                                                  Microsoft SQL Server

                                                                                                                                                  Postgres JDBC driver 42.2.5

                                                                                                                                                  SQL Server JDBC driver 4.6

                                                                                                                                                  This driver version supports SQL Server Always On ( Always On Failover Cluster Instances and Always On availability groups).

                                                                                                                                                  Additional requirements for Windows authentication against Microsoft SQL Server

                                                                                                                                                  If you want HDS nodes to use Windows authentication to gain access to your keystore database on Microsoft SQL Server, then you need the following configuration in your environment:

                                                                                                                                                  • The HDS nodes, Active Directory infrastructure, and MS SQL Server must all be synchronized with NTP.

                                                                                                                                                  • The Windows account you provide to HDS nodes must have read/write access to the database.

                                                                                                                                                  • The DNS servers you provide to HDS nodes must be able to resolve your Key Distribution Center (KDC).

                                                                                                                                                  • You may register the HDS database instance on your Microsoft SQL Server as a Service Principal Name (SPN) on your Active Directory. See Register a Service Principal Name for Kerberos Connections.

                                                                                                                                                    The HDS setup tool, HDS launcher, and local KMS all need to use Windows authentication to access the keystore database. They use the details from your ISO configuration to construct the SPN when requesting access with Kerberos authentication.

                                                                                                                                                  External connectivity requirements

                                                                                                                                                  Configure your firewall to allow the following connectivity for the HDS applications:

                                                                                                                                                  Application

                                                                                                                                                  Protocol

                                                                                                                                                  Port

                                                                                                                                                  Direction from App

                                                                                                                                                  Destination

                                                                                                                                                  Hybrid Data Security nodes

                                                                                                                                                  TCP

                                                                                                                                                  443

                                                                                                                                                  Outbound HTTPS and WSS

                                                                                                                                                  • Webex servers:

                                                                                                                                                    • *.wbx2.com

                                                                                                                                                    • *.ciscospark.com

                                                                                                                                                  • All Common Identity hosts

                                                                                                                                                  • Other URLs that are listed for Hybrid Data Security in the Additional URLs for Webex Hybrid Services table of Network Requirements for Webex Services

                                                                                                                                                  HDS Setup Tool

                                                                                                                                                  TCP

                                                                                                                                                  443

                                                                                                                                                  Outbound HTTPS

                                                                                                                                                  • *.wbx2.com

                                                                                                                                                  • All Common Identity hosts

                                                                                                                                                  • hub.docker.com

                                                                                                                                                  The Hybrid Data Security nodes work with network access translation (NAT) or behind a firewall, as long as the NAT or firewall allows the required outbound connections to the domain destinations in the preceding table. For connections going inbound to the Hybrid Data Security nodes, no ports should be visible from the internet. Within your data center, clients need access to the Hybrid Data Security nodes on TCP ports 443 and 22, for administrative purposes.

                                                                                                                                                  The URLs for the Common Identity (CI) hosts are region-specific. These are the current CI hosts:

                                                                                                                                                  Region

                                                                                                                                                  Common Identity Host URLs

                                                                                                                                                  Americas

                                                                                                                                                  • https://idbroker.webex.com

                                                                                                                                                  • https://identity.webex.com

                                                                                                                                                  • https://idbroker-b-us.webex.com

                                                                                                                                                  • https://identity-b-us.webex.com

                                                                                                                                                  European Union

                                                                                                                                                  • https://idbroker-eu.webex.com

                                                                                                                                                  • https://identity-eu.webex.com

                                                                                                                                                  Canada

                                                                                                                                                  • https://idbroker-ca.webex.com

                                                                                                                                                  • https://identity-ca.webex.com

                                                                                                                                                  Proxy Server Requirements

                                                                                                                                                  • We officially support the following proxy solutions that can integrate with your Hybrid Data Security nodes.

                                                                                                                                                    • Transparent proxy—Cisco Web Security Appliance (WSA).

                                                                                                                                                    • Explicit proxy—Squid.

                                                                                                                                                      Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:) connections. To work around this issue, see Configure Squid Proxies for Hybrid Data Security.

                                                                                                                                                  • We support the following authentication type combinations for explicit proxies:

                                                                                                                                                    • No authentication with HTTP or HTTPS

                                                                                                                                                    • Basic authentication with HTTP or HTTPS

                                                                                                                                                    • Digest authentication with HTTPS only

                                                                                                                                                  • For a transparent inspecting proxy or an HTTPS explicit proxy, you must have a copy of the proxy's root certificate. The deployment instructions in this guide tell you how to upload the copy to the Hybrid Data Security nodes' trust stores.

                                                                                                                                                  • The network hosting the HDS nodes must be configured to force outbound TCP traffic on port 443 to route through the proxy.

                                                                                                                                                  • Proxies that inspect web traffic may interfere with web socket connections. If this problem occurs, bypassing (not inspecting) traffic to wbx2.com and ciscospark.com will solve the problem.

                                                                                                                                                  Complete the Prerequisites for Hybrid Data Security

                                                                                                                                                  Use this checklist to ensure that you are ready to install and configure your Hybrid Data Security cluster.
                                                                                                                                                  1

                                                                                                                                                  Make sure your Webex organization is enabled for Pro Pack for Cisco Webex Control Hub, and get the credentials of an account with full organization administrator rights. Contact your Cisco partner or account manager for help with this process.

                                                                                                                                                  2

                                                                                                                                                  Choose a domain name for your HDS deployment (for example, hds.company.com) and obtain a certificate chain containing an X.509 certificate, private key, and any intermediate certificates. The certificate chain must meet the requirements in X.509 Certificate Requirements.

                                                                                                                                                  3

                                                                                                                                                  Prepare identical virtual hosts that you will set up as Hybrid Data Security nodes in your cluster. You need at least two separate hosts (3 recommended) colocated in the same secure data center, which meet the requirements in Virtual Host Requirements.

                                                                                                                                                  4

                                                                                                                                                  Prepare the database server that will act as the key data store for the cluster, according to the Database server requirements. The database server must be colocated in the secure data center with the virtual hosts.

                                                                                                                                                  1. Create a database for key storage. (You must create this database—do not use the default database. The HDS applications, when installed, create the database schema.)

                                                                                                                                                  2. Gather the details that the nodes will use to communicate with the database server:

                                                                                                                                                    • the host name or IP address (host) and port

                                                                                                                                                    • the name of the database (dbname) for key storage

                                                                                                                                                    • the username and password of a user with all privileges on the key storage database

                                                                                                                                                  5

                                                                                                                                                  For quick disaster recovery, set up a backup environment in a different data center. The backup environment mirrors the production environment of VMs and a backup database server. For example, if production has 3 VMs running HDS nodes, the backup environment should have 3 VMs.

                                                                                                                                                  6

                                                                                                                                                  Set up a syslog host to collect logs from the nodes in the cluster. Gather its network address and syslog port (default is UDP 514).

                                                                                                                                                  7

                                                                                                                                                  Create a secure backup policy for the Hybrid Data Security nodes, the database server, and the syslog host. At a minimum, to prevent unrecoverable data loss, you must back up the database and the configuration ISO file generated for the Hybrid Data Security nodes.

                                                                                                                                                  Because the Hybrid Data Security nodes store the keys used in encryption and decryption of content, failure to maintain an operational deployment will result in the UNRECOVERABLE LOSS of that content.

                                                                                                                                                  Webex App clients cache their keys, so an outage may not be immediately noticeable but will become evident over time. While temporary outages are impossible to prevent, they are recoverable. However, complete loss (no backups available) of either the database or configuration ISO file will result in unrecoverable customer data. The operators of the Hybrid Data Security nodes are expected to maintain frequent backups of the database and the configuration ISO file, and be prepared to rebuild the Hybrid Data Security data center if a catastrophic failure occurs.

                                                                                                                                                  8

                                                                                                                                                  Ensure that your firewall configuration allows connectivity for your Hybrid Data Security nodes as outlined in External connectivity requirements.

                                                                                                                                                  9

                                                                                                                                                  Install Docker ( https://www.docker.com) on any local machine running a supported OS (Microsoft Windows 10 Professional or Enterprise 64-bit, or Mac OSX Yosemite 10.10.3 or above) with a web browser that can access it at http://127.0.0.1:8080.

                                                                                                                                                  You use the Docker instance to download and run the HDS Setup Tool, which builds the local configuration information for all the Hybrid Data Security nodes. Your organization might need a Docker Desktop license. See Docker Desktop Requirements for more information.

                                                                                                                                                  To install and run the HDS Setup Tool, the local machine must have the connectivity outlined in External connectivity requirements.

                                                                                                                                                  10

                                                                                                                                                  If you're integrating a proxy with Hybrid Data Security, make sure that it meets the Proxy Server Requirements.

                                                                                                                                                  11

                                                                                                                                                  If your organization uses directory synchronization, create a group in Active Directory called HdsTrialGroup, and add pilot users. The trial group can have up to 250 users. The HdsTrialGroup object must be synchronized to the cloud before you can start a trial for your organization. To synchronize a group object, select it in the Directory Connector's Configuration > Object Selection menu. (For detailed instructions see the Deployment Guide for Cisco Directory Connector.)

                                                                                                                                                  Keys for a given space are set by the creator of the space. When selecting pilot users, bear in mind that if you decide to permanently deactivate the Hybrid Data Security deployment, all users lose access to content in the spaces that were created by the pilot users. The loss becomes apparent as soon as users' apps refresh their cached copies of the content.

                                                                                                                                                  Set up a Hybrid Data Security Cluster

                                                                                                                                                  Hybrid Data Security Deployment Task Flow

                                                                                                                                                  Before you begin

                                                                                                                                                  Prepare Your Environment

                                                                                                                                                  1

                                                                                                                                                  Perform initial set up and download installation files

                                                                                                                                                  Download the OVA file to your local machine for later use.

                                                                                                                                                  2

                                                                                                                                                  Create a Configuration ISO for the HDS Hosts

                                                                                                                                                  Use the HDS Setup Tool to create an ISO configuration file for the Hybrid Data Security nodes.

                                                                                                                                                  3

                                                                                                                                                  Install the HDS Host OVA

                                                                                                                                                  Create a virtual machine from the OVA file and perform initial configuration, such as network settings.

                                                                                                                                                  The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions.

                                                                                                                                                  4

                                                                                                                                                  Set up the Hybrid Data Security VM

                                                                                                                                                  Sign in to the VM console and set the sign-in credentials. Configure the network settings for the node if you didn't configure them at the time of OVA deployment.

                                                                                                                                                  5

                                                                                                                                                  Upload and Mount the HDS Configuration ISO

                                                                                                                                                  Configure the VM from the ISO configuration file that you created with the HDS Setup Tool.

                                                                                                                                                  6

                                                                                                                                                  Configure the HDS Node for Proxy Integration

                                                                                                                                                  If the network environment requires proxy configuration, specify the type of proxy that you will use for the node, and add the proxy certificate to the trust store if needed.

                                                                                                                                                  7

                                                                                                                                                  Register the First Node in the Cluster

                                                                                                                                                  Register the VM with the Cisco Webex cloud as a Hybrid Data Security node.

                                                                                                                                                  8

                                                                                                                                                  Create and Register More Nodes

                                                                                                                                                  Complete the cluster setup.

                                                                                                                                                  9

                                                                                                                                                  Run a Trial and Move to Production (next chapter)

                                                                                                                                                  Until you start a trial, your nodes generate an alarm indicating that your service is not yet activated.

                                                                                                                                                  Download Installation Files

                                                                                                                                                  In this task, you download an OVA file to your machine (not to the servers you set up as Hybrid Data Security nodes). You use this file later in the installation process.
                                                                                                                                                  1

                                                                                                                                                  Sign in to https://admin.webex.com, and then click Services.

                                                                                                                                                  2

                                                                                                                                                  In the Hybrid Services section, find the Hybrid Data Security card, and then click Set up.

                                                                                                                                                  If the card is disabled or you don’t see it, contact your account team or your partner organization. Give them your account number and ask to enable your organization for Hybrid Data Security. To find the account number, click the gear at the top right, next to your organization name.

                                                                                                                                                  You can also download the OVA at any time from the Help section on the Settings page. On the Hybrid Data Security card, click Edit settings to open the page. Then, click Download Hybrid Data Security software in the Help section.

                                                                                                                                                  Older versions of the software package (OVA) will not be compatible with the latest Hybrid Data Security upgrades. This can result in issues while upgrading the application. Make sure you download the latest version of the OVA file.

                                                                                                                                                  3

                                                                                                                                                  Select No to indicate that you haven’t set up the node yet, and then click Next.

                                                                                                                                                  The OVA file automatically begins to download. Save the file to a location on your machine.
                                                                                                                                                  4

                                                                                                                                                  Optionally, click Open Deployment Guide to check if there’s a later version of this guide available.

                                                                                                                                                  Create a Configuration ISO for the HDS Hosts

                                                                                                                                                  The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.

                                                                                                                                                  Before you begin

                                                                                                                                                  • The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Control Hub account with full administrator rights for your organization.

                                                                                                                                                    If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5. This table gives some possible environment variables:

                                                                                                                                                    Description

                                                                                                                                                    Variable

                                                                                                                                                    HTTP Proxy without authentication

                                                                                                                                                    GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT

                                                                                                                                                    HTTPS Proxy without authentication

                                                                                                                                                    GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT

                                                                                                                                                    HTTP Proxy with authentication

                                                                                                                                                    GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT

                                                                                                                                                    HTTPS Proxy with authentication

                                                                                                                                                    GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT

                                                                                                                                                  • The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:

                                                                                                                                                    • Database credentials

                                                                                                                                                    • Certificate updates

                                                                                                                                                    • Changes to authorization policy

                                                                                                                                                  • If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.

                                                                                                                                                  1

                                                                                                                                                  At your machine's command line, enter the appropriate command for your environment:

                                                                                                                                                  In regular environments:

                                                                                                                                                  docker rmi ciscocitg/hds-setup:stable

                                                                                                                                                  In FedRAMP environments:

                                                                                                                                                  docker rmi ciscocitg/hds-setup-fedramp:stable

                                                                                                                                                  This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.

                                                                                                                                                  2

                                                                                                                                                  To sign in to the Docker image registry, enter the following:

                                                                                                                                                  docker login -u hdscustomersro
                                                                                                                                                  3

                                                                                                                                                  At the password prompt, enter this hash:

                                                                                                                                                  dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo
                                                                                                                                                  4

                                                                                                                                                  Download the latest stable image for your environment:

                                                                                                                                                  In regular environments:

                                                                                                                                                  docker pull ciscocitg/hds-setup:stable

                                                                                                                                                  In FedRAMP environments:

                                                                                                                                                  docker pull ciscocitg/hds-setup-fedramp:stable
                                                                                                                                                  5

                                                                                                                                                  When the pull completes, enter the appropriate command for your environment:

                                                                                                                                                  • In regular environments without a proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable
                                                                                                                                                  • In regular environments with an HTTP proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable
                                                                                                                                                  • In regular environments with an HTTPS proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable
                                                                                                                                                  • In FedRAMP environments without a proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable
                                                                                                                                                  • In FedRAMP environments with an HTTP proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable
                                                                                                                                                  • In FedRAMP environments with an HTTPS proxy:

                                                                                                                                                    docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable

                                                                                                                                                  When the container is running, you see "Express server listening on port 8080."

                                                                                                                                                  6

                                                                                                                                                  The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost.

                                                                                                                                                  Use a web browser to go to the localhost, http://127.0.0.1:8080, and enter customer admin username for Control Hub at the prompt.

                                                                                                                                                  The tool uses this first entry of the username to set the proper environment for that account. The tool then displays the standard sign-in prompt.

                                                                                                                                                  7

                                                                                                                                                  When prompted, enter your Control Hub customer admin sign-in credentials, and then click Log in to allow access to the required services for Hybrid Data Security.

                                                                                                                                                  8

                                                                                                                                                  On the Setup Tool overview page, click Get Started.

                                                                                                                                                  9

                                                                                                                                                  On the ISO Import page, you have these options:

                                                                                                                                                  • No—If you’re creating your first HDS node, you don't have an ISO file to upload.
                                                                                                                                                  • Yes—If you already created HDS nodes, then you select your ISO file in the browse and upload it.
                                                                                                                                                  10

                                                                                                                                                  Check that your X.509 certificate meets the requirements in X.509 Certificate Requirements.

                                                                                                                                                  • If you never uploaded a certificate before, upload the X.509 certificate, enter the password, and click Continue.
                                                                                                                                                  • If your certificate is OK, click Continue.
                                                                                                                                                  • If your certificate has expired or you want to replace it, select No for Continue using HDS certificate chain and private key from previous ISO?. Upload a new X.509 certificate, enter the password, and click Continue.
                                                                                                                                                  11

                                                                                                                                                  Enter the database address and account for HDS to access your key datastore:

                                                                                                                                                  1. Select your Database Type (PostgreSQL or Microsoft SQL Server).

                                                                                                                                                    If you choose Microsoft SQL Server, you get an Authentication Type field.

                                                                                                                                                  2. (Microsoft SQL Server only) Select your Authentication Type:

                                                                                                                                                    • Basic Authentication: You need a local SQL Server account name in the Username field.

                                                                                                                                                    • Windows Authentication: You need a Windows account in the format username@DOMAIN in the Username field.

                                                                                                                                                  3. Enter the database server address in the form <hostname>:<port> or <IP-address>:<port>.

                                                                                                                                                    Example:
                                                                                                                                                    dbhost.example.org:1433 or 198.51.100.17:1433

                                                                                                                                                    You can use an IP address for basic authentication, if the nodes can't use DNS to resolve the hostname.

                                                                                                                                                    If you are using Windows authentication, you must enter a Fully Qualified Domain Name in the format dbhost.example.org:1433

                                                                                                                                                  4. Enter the Database Name.

                                                                                                                                                  5. Enter the Username and Password of a user with all privileges on the key storage database.

                                                                                                                                                  12

                                                                                                                                                  Select a TLS Database Connection Mode:

                                                                                                                                                  Mode

                                                                                                                                                  Description

                                                                                                                                                  Prefer TLS (default option)

                                                                                                                                                  HDS nodes don’t require TLS to connect to the database server. If you enable TLS on the database server, the nodes attempt an encrypted connection.

                                                                                                                                                  Require TLS

                                                                                                                                                  HDS nodes connect only if the database server can negotiate TLS.

                                                                                                                                                  Require TLS and verify certificate signer

                                                                                                                                                  This mode isn’t applicable for SQL Server databases.

                                                                                                                                                  • HDS nodes connect only if the database server can negotiate TLS.

                                                                                                                                                  • After establishing a TLS connection, the node compares the signer of the certificate from the database server to the certificate authority in the Database root certificate. If they don't match, the node drops the connection.

                                                                                                                                                  Use the Database root certificate control below the drop-down to upload the root certificate for this option.

                                                                                                                                                  Require TLS and verify certificate signer and hostname

                                                                                                                                                  • HDS nodes connect only if the database server can negotiate TLS.

                                                                                                                                                  • After establishing a TLS connection, the node compares the signer of the certificate from the database server to the certificate authority in the Database root certificate. If they don't match, the node drops the connection.

                                                                                                                                                  • The nodes also verify that the hostname in the server certificate matches the hostname in the Database host and port field. The names must match exactly, or the node drops the connection.

                                                                                                                                                  Use the Database root certificate control below the drop-down to upload the root certificate for this option.

                                                                                                                                                  When you upload the root certificate (if necessary) and click Continue, the HDS Setup Tool tests the TLS connection to the database server. The tool also verifies the certificate signer and hostname, if applicable. If a test fails, the tool shows an error message describing the problem. You can choose whether to ignore the error and continue with the setup. (Because of connectivity differences, the HDS nodes might be able to establish the TLS connection even if the HDS Setup Tool machine can't successfully test it.)

                                                                                                                                                  13

                                                                                                                                                  On the System Logs page, configure your Syslogd server:

                                                                                                                                                  1. Enter the syslog server URL.

                                                                                                                                                    If the server isn’t DNS-resolvable from the nodes for your HDS cluster, use an IP address in the URL.

                                                                                                                                                    Example:
                                                                                                                                                    udp://10.92.43.23:514 indicates logging to Syslogd host 10.92.43.23 on UDP port 514.
                                                                                                                                                  2. If you set up your server to use TLS encryption, check Is your syslog server configured for SSL encryption?.

                                                                                                                                                    If you check this check box, make sure you enter a TCP URL such as tcp://10.92.43.23:514.

                                                                                                                                                  3. From the Choose syslog record termination drop-down, choose the appropriate setting for your ISO file: Choose or Newline is used for Graylog and Rsyslog TCP

                                                                                                                                                    • Null byte -- \x00

                                                                                                                                                    • Newline -- \n—Select this choice for Graylog and Rsyslog TCP.

                                                                                                                                                  4. Click Continue.

                                                                                                                                                  14

                                                                                                                                                  (Optional) You can change the default value for some database connection parameters in Advanced Settings. Generally, this parameter is the only one that you might want to change:

                                                                                                                                                  app_datasource_connection_pool_maxSize: 10
                                                                                                                                                  15

                                                                                                                                                  Click Continue on the Reset Service Accounts Password screen.

                                                                                                                                                  Service account passwords have a nine-month lifespan. Use this screen when your passwords are nearing expiry or you want to reset them to invalidate previous ISO files.

                                                                                                                                                  16

                                                                                                                                                  Click Download ISO File. Save the file in a location that's easy to find.

                                                                                                                                                  17

                                                                                                                                                  Make a backup copy of the ISO file on your local system.

                                                                                                                                                  Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes.

                                                                                                                                                  18

                                                                                                                                                  To shut down the Setup tool, type CTRL+C.

                                                                                                                                                  What to do next

                                                                                                                                                  Back up the configuration ISO file. You need it to create more nodes for recovery, or to make configuration changes. If you lose all copies of the ISO file, you've also lost the master key. Recovering the keys from your PostgreSQL or Microsoft SQL Server database isn't possible.

                                                                                                                                                  We never have a copy of this key and can't help if you lose it.

                                                                                                                                                  Install the HDS Host OVA

                                                                                                                                                  Use this procedure to create a virtual machine from the OVA file.
                                                                                                                                                  1

                                                                                                                                                  Use the VMware vSphere client on your computer to log into the ESXi virtual host.

                                                                                                                                                  2

                                                                                                                                                  Select File > Deploy OVF Template.

                                                                                                                                                  3

                                                                                                                                                  In the wizard, specify the location of the OVA file that you downloaded earlier, and then click Next.

                                                                                                                                                  4

                                                                                                                                                  On the Select a name and folder page, enter a Virtual machine name for the node (for example, "HDS_Node_1"), choose a location where the virtual machine node deployment can reside, and then click Next.

                                                                                                                                                  5

                                                                                                                                                  On the Select a compute resource page, choose the destination compute resource, and then click Next.

                                                                                                                                                  A validation check runs. After it finishes, the template details appear.

                                                                                                                                                  6

                                                                                                                                                  Verify the template details and then click Next.

                                                                                                                                                  7

                                                                                                                                                  If you are asked to choose the resource configuration on the Configuration page, click 4 CPU and then click Next.

                                                                                                                                                  8

                                                                                                                                                  On the Select storage page, click Next to accept the default disk format and VM storage policy.

                                                                                                                                                  9

                                                                                                                                                  On the Select networks page, choose the network option from the list of entries to provide the desired connectivity to the VM.

                                                                                                                                                  10

                                                                                                                                                  On the Customize template page, configure the following network settings:

                                                                                                                                                  • Hostname—Enter the FQDN (hostname and domain) or a single word hostname for the node.
                                                                                                                                                    • You do not need to set the domain to match the domain that you used to obtain the X.509 certificate.

                                                                                                                                                    • To ensure a successful registration to the cloud, use only lowercase characters in the FQDN or hostname that you set for the node. Capitalization is not supported at this time.

                                                                                                                                                    • The total length of the FQDN must not exceed 64 characters.

                                                                                                                                                  • IP Address— Enter the IP address for the internal interface of the node.

                                                                                                                                                    Your node should have an internal IP address and DNS name. DHCP is not supported.

                                                                                                                                                  • Mask—Enter the subnet mask address in dot-decimal notation. For example, 255.255.255.0.
                                                                                                                                                  • Gateway—Enter the gateway IP address. A gateway is a network node that serves as an access point to another network.
                                                                                                                                                  • DNS Servers—Enter a comma-separated list of DNS servers, which handle translating domain names to numeric IP addresses. (Up to 4 DNS entries are allowed.)
                                                                                                                                                  • NTP Servers—Enter your organization's NTP server or another external NTP server that can be used in your organization. The default NTP servers may not work for all enterprises. You can also use a comma-separated list to enter multiple NTP servers.
                                                                                                                                                  • Deploy all the nodes on the same subnet or VLAN, so that all nodes in a cluster are reachable from clients in your network for administrative purposes.

                                                                                                                                                  If preferred, you can skip the network setting configuration and follow the steps in Set up the Hybrid Data Security VM to configure the settings from the node console.

                                                                                                                                                  The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions.

                                                                                                                                                  11

                                                                                                                                                  Right-click the node VM, and then choose Power > Power On.

                                                                                                                                                  The Hybrid Data Security software is installed as a guest on the VM Host. You are now ready to sign in to the console and configure the node.

                                                                                                                                                  Troubleshooting Tips

                                                                                                                                                  You may experience a delay of a few minutes before the node containers come up. A bridge firewall message appears on the console during first boot, during which you can't sign in.

                                                                                                                                                  Set up the Hybrid Data Security VM

                                                                                                                                                  Use this procedure to sign in to the Hybrid Data Security node VM console for the first time and set the sign-in credentials. You can also use the console to configure the network settings for the node if you didn't configure them at the time of OVA deployment.

                                                                                                                                                  1

                                                                                                                                                  In the VMware vSphere client, select your Hybrid Data Security node VM and select the Console tab.

                                                                                                                                                  The VM boots up and a login prompt appears. If the login prompt does not display, press Enter.
                                                                                                                                                  2

                                                                                                                                                  Use the following default login and password to sign in and change the credentials:

                                                                                                                                                  1. Login: admin

                                                                                                                                                  2. Password: cisco

                                                                                                                                                  Since you are signing in to your VM for the first time, you are required to change the administrator password.

                                                                                                                                                  3

                                                                                                                                                  If you already configured the network settings in Install the HDS Host OVA, skip the rest of this procedure. Otherwise, in the main menu, select the Edit Configuration option.

                                                                                                                                                  4

                                                                                                                                                  Set up a static configuration with IP address, Mask, Gateway and DNS information. Your node should have an internal IP address and DNS name. DHCP is not supported.

                                                                                                                                                  5

                                                                                                                                                  (Optional) Change the hostname, domain or NTP server(s), if needed to match your network policy.

                                                                                                                                                  You do not need to set the domain to match the domain that you used to obtain the X.509 certificate.

                                                                                                                                                  6

                                                                                                                                                  Save the network configuration and reboot the VM so that the changes take effect.

                                                                                                                                                  Upload and Mount the HDS Configuration ISO

                                                                                                                                                  Use this procedure to configure the virtual machine from the ISO file that you created with the HDS Setup Tool.

                                                                                                                                                  Before you begin

                                                                                                                                                  Because the ISO file holds the master key, it should only be exposed on a "need to know" basis, for access by the Hybrid Data Security VMs and any administrators who might need to make changes. Make sure that only those administrators can access the datastore.

                                                                                                                                                  1

                                                                                                                                                  Upload the ISO file from your computer:

                                                                                                                                                  1. In the VMware vSphere client's left navigation pane, click on the ESXi server.

                                                                                                                                                  2. On the Configuration tab's Hardware list, click Storage.

                                                                                                                                                  3. In the Datastores list, right-click on the datastore for your VMs and click Browse Datastore.

                                                                                                                                                  4. Click on the Upload Files icon, and then click Upload File.

                                                                                                                                                  5. Browse to the location where you downloaded the ISO file on your computer and click Open.

                                                                                                                                                  6. Click Yes to accept the upload/download operation warning, and close the datastore dialog.

                                                                                                                                                  2

                                                                                                                                                  Mount the ISO file:

                                                                                                                                                  1. In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.

                                                                                                                                                  2. Click OK to accept the restricted edit options warning.

                                                                                                                                                  3. Click CD/DVD Drive 1, select the option to mount from a datastore ISO file, and browse to the location where you uploaded the configuration ISO file.

                                                                                                                                                  4. Check Connected and Connect at power on.

                                                                                                                                                  5. Save your changes and reboot the virtual machine.

                                                                                                                                                  What to do next

                                                                                                                                                  If your IT policy requires, you can optionally unmount the ISO file after all your nodes pick up the configuration changes. See (Optional) Unmount ISO After HDS Configuration for details.

                                                                                                                                                  Configure the HDS Node for Proxy Integration

                                                                                                                                                  If the network environment requires a proxy, use this procedure to specify the type of proxy that you want to integrate with Hybrid Data Security. If you choose a transparent inspecting proxy or an HTTPS explicit proxy, you can use the node's interface to upload and install the root certificate. You can also check the proxy connection from the interface, and troubleshoot any potential issues.

                                                                                                                                                  Before you begin

                                                                                                                                                  1

                                                                                                                                                  Enter the HDS node setup URL https://[HDS Node IP or FQDN]/setup in a web browser, enter the admin credentials that you set up for the node, and then click Sign In.

                                                                                                                                                  2

                                                                                                                                                  Go to Trust Store & Proxy, and then choose an option:

                                                                                                                                                  • No Proxy—The default option before you integrate a proxy. No certificate update is required.
                                                                                                                                                  • Transparent Non-Inspecting Proxy—Nodes are not configured to use a specific proxy server address and should not require any changes to work with a non-inspecting proxy. No certificate update is required.
                                                                                                                                                  • Transparent Inspecting Proxy—Nodes are not configured to use a specific proxy server address. No HTTPS configuration changes are necessary on the Hybrid Data Security deployment, however, the HDS nodes need a root certificate so that they trust the proxy. Inspecting proxies are typically used by IT to enforce policies on which websites can be visited and which types of content are not permitted. This type of proxy decrypts all your traffic (even HTTPS).
                                                                                                                                                  • Explicit Proxy—With explicit proxy, you tell the client (HDS nodes) which proxy server to use, and this option supports several authentication types. After you choose this option, you must enter the following information:
                                                                                                                                                    1. Proxy IP/FQDN—Address that can be used to reach the proxy machine.

                                                                                                                                                    2. Proxy Port—A port number that the proxy uses to listen for proxied traffic.

                                                                                                                                                    3. Proxy Protocol—Choose http (views and controls all requests that are received from the client) or https (provides a channel to the server and the client receives and validates the server's certificate). Choose an option based on what your proxy server supports.

                                                                                                                                                    4. Authentication Type—Choose from among the following authentication types:

                                                                                                                                                      • None—No further authentication is required.

                                                                                                                                                        Available for HTTP or HTTPS proxies.

                                                                                                                                                      • Basic—Used for an HTTP User Agent to provide a user name and password when making a request. Uses Base64 encoding.

                                                                                                                                                        Available for HTTP or HTTPS proxies.

                                                                                                                                                        If you choose this option, you must also enter the user name and password.

                                                                                                                                                      • Digest—Used to confirm the account before sending sensitive information. Applies a hash function on the user name and password before sending over the network.

                                                                                                                                                        Available for HTTPS proxies only.

                                                                                                                                                        If you choose this option, you must also enter the user name and password.

                                                                                                                                                  Follow the next steps for a transparent inspecting proxy, an HTTP explicit proxy with Basic authentication, or an HTTPS explicit proxy.

                                                                                                                                                  3

                                                                                                                                                  Click Upload a Root Certificate or End Entity Certificate, and then navigate to a choose the root certificate for the proxy.

                                                                                                                                                  The certificate is uploaded but not yet installed because you must reboot the node to install the certificate. Click the chevron arrow by the certificate issuer name to get more details or click Delete if you made a mistake and want to reupload the file.

                                                                                                                                                  4

                                                                                                                                                  Click Check Proxy Connection to test the network connectivity between the node and the proxy.

                                                                                                                                                  If the connection test fails, you'll see an error message that shows the reason and how you can correct the issue.

                                                                                                                                                  If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server. This condition is expected in many explicit proxy configurations. You can continue with the setup, and the node will function in Blocked External DNS Resolution mode. If you think this is an error, complete these steps, and then see Turn off Blocked External DNS Resolution Mode.

                                                                                                                                                  5

                                                                                                                                                  After the connection test passes, for explicit proxy set to https only, turn the toggle on to Route all port 443/444 https requests from this node through the explicit proxy. This setting requires 15 seconds to take effect.

                                                                                                                                                  6

                                                                                                                                                  Click Install All Certificates Into the Trust Store (appears for an HTTPS explicit proxy or a transparent inspecting proxy) or Reboot (appears for an HTTP explicit proxy), read the prompt, and then click Install if you're ready.

                                                                                                                                                  The node reboots within a few minutes.

                                                                                                                                                  7

                                                                                                                                                  After the node reboots, sign in again if needed, and then open the Overview page to check the connectivity checks to make sure they are all in green status.

                                                                                                                                                  The proxy connection check only tests a subdomain of webex.com. If there are connectivity problems, a common issue is that some of the cloud domains listed in the install instructions are being blocked at the proxy.

                                                                                                                                                  Register the First Node in the Cluster

                                                                                                                                                  This task takes the generic node that you created in the Set up the Hybrid Data Security VM, registers the node with the Webex cloud, and turns it into a Hybrid Data Security node.

                                                                                                                                                  When you register your first node, you create a cluster to which the node is assigned. A cluster contains one or more nodes deployed to provide redundancy.

                                                                                                                                                  Before you begin

                                                                                                                                                  • Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.

                                                                                                                                                  • Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.

                                                                                                                                                  1

                                                                                                                                                  Sign in to https://admin.webex.com.

                                                                                                                                                  2

                                                                                                                                                  From the menu on the left side of the screen, select Services.

                                                                                                                                                  3

                                                                                                                                                  In the Hybrid Services section, find Hybrid Data Security and click Set up.

                                                                                                                                                  The Register Hybrid Data Security Node page appears.
                                                                                                                                                  4

                                                                                                                                                  Select Yes to indicate that you have set up the node and are ready to register it, and then click Next.

                                                                                                                                                  5

                                                                                                                                                  In the first field, enter a name for the cluster to which you want to assign your Hybrid Data Security node.

                                                                                                                                                  We recommend that you name a cluster based on where the nodes of the cluster are located geographically. Examples: "San Francisco" or "New York" or "Dallas"

                                                                                                                                                  6

                                                                                                                                                  In the second field, enter the internal IP address or fully qualified domain name (FQDN) of your node and click Next.

                                                                                                                                                  This IP address or FQDN should match the IP address or hostname and domain that you used in Set up the Hybrid Data Security VM.

                                                                                                                                                  A message appears indicating you can register your node to the Webex.
                                                                                                                                                  7

                                                                                                                                                  Click Go to Node.

                                                                                                                                                  8

                                                                                                                                                  Click Continue in the warning message.

                                                                                                                                                  After a few moments, you are redirected to the node connectivity tests for Webex services. If all tests are successful, the Allow Access to Hybrid Data Security Node page appears. There, you confirm that you want to give permissions to your Webex organization to access your node.
                                                                                                                                                  9

                                                                                                                                                  Check the Allow Access to Your Hybrid Data Security Node checkbox, and then click Continue.

                                                                                                                                                  Your account is validated and the "Registration Complete" message indicates that your node is now registered to the Webex cloud.
                                                                                                                                                  10

                                                                                                                                                  Click the link or close the tab to go back to the Control Hub Hybrid Data Security page.

                                                                                                                                                  On the Hybrid Data Security page, the new cluster containing the node that you registered is displayed. The node will automatically download the latest software from the cloud.

                                                                                                                                                  Create and Register More Nodes

                                                                                                                                                  To add additional nodes to your cluster, you simply create additional VMs and mount the same configuration ISO file, then register the node. We recommend that you have at least 3 nodes.

                                                                                                                                                  At this time, the backup VMs that you created in Complete the Prerequisites for Hybrid Data Security are standby hosts which are only used in the event of disaster recovery; they are not registered with the system until then. For details, see Disaster Recovery using Standby Data Center.

                                                                                                                                                  Before you begin

                                                                                                                                                  • Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.

                                                                                                                                                  • Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.

                                                                                                                                                  1

                                                                                                                                                  Create a new virtual machine from the OVA, repeating the steps in Install the HDS Host OVA.

                                                                                                                                                  2

                                                                                                                                                  Set up the initial configuration on the new VM, repeating the steps in Set up the Hybrid Data Security VM.

                                                                                                                                                  3

                                                                                                                                                  On the new VM, repeat the steps in Upload and Mount the HDS Configuration ISO.

                                                                                                                                                  4

                                                                                                                                                  If you are setting up a proxy for your deployment, repeat the steps in Configure the HDS Node for Proxy Integration as needed for the new node.

                                                                                                                                                  5

                                                                                                                                                  Register the node.

                                                                                                                                                  1. In https://admin.webex.com, select Services from the menu on the left side of the screen.

                                                                                                                                                  2. In the Hybrid Services section, find the Hybrid Data Security card and click Resources.

                                                                                                                                                    The Hybrid Data Security Resources page appears.
                                                                                                                                                  3. Click Add Resource.

                                                                                                                                                  4. In the first field, select the name of your existing cluster.

                                                                                                                                                  5. In the second field, enter the internal IP address or fully qualified domain name (FQDN) of your node and click Next.

                                                                                                                                                    A message appears indicating you can register your node to the Webex cloud.
                                                                                                                                                  6. Click Go to Node.

                                                                                                                                                    After a few moments, you are redirected to the node connectivity tests for Webex services. If all tests are successful, the Allow Access to Hybrid Data Security Node page appears. There, you confirm that you want to give permissions to your organization to access your node.
                                                                                                                                                  7. Check the Allow Access to Your Hybrid Data Security Node checkbox, and then click Continue.

                                                                                                                                                    Your account is validated and the "Registration Complete" message indicates that your node is now registered to the Webex cloud.
                                                                                                                                                  8. Click the link or close the tab to go back to the Control Hub Hybrid Data Security page.

                                                                                                                                                  Your node is registered. Note that until you start a trial, your nodes generate an alarm indicating that your service is not yet activated.

                                                                                                                                                  What to do next

                                                                                                                                                  Run a Trial and Move to Production (next chapter)

                                                                                                                                                  Run a Trial and Move to Production

                                                                                                                                                  Trial to Production Task Flow

                                                                                                                                                  After you set up a Hybrid Data Security cluster, you can start a pilot, add users to it, and begin using it for testing and verifying your deployment in preparation for moving to production.

                                                                                                                                                  1

                                                                                                                                                  If applicable, synchronize the HdsTrialGroup group object.

                                                                                                                                                  If your organization uses directory synchronization for users, you must select the HdsTrialGroup group object for synchronization to the cloud before you can start a trial. For instructions, see the Deployment Guide for Cisco Directory Connector.

                                                                                                                                                  2

                                                                                                                                                  Activate Trial

                                                                                                                                                  Start a trial. Until you do this task, your nodes generate an alarm indicating that the service is not yet activated.

                                                                                                                                                  3

                                                                                                                                                  Test Your Hybrid Data Security Deployment

                                                                                                                                                  Check that key requests are passing to your Hybrid Data Security deployment.

                                                                                                                                                  4

                                                                                                                                                  Monitor Hybrid Data Security Health

                                                                                                                                                  Check status, and set up email notifications for alarms.

                                                                                                                                                  5

                                                                                                                                                  Add or Remove Users from Your Trial

                                                                                                                                                  6

                                                                                                                                                  Complete the trial phase with one of the following actions:

                                                                                                                                                  Activate Trial

                                                                                                                                                  Before you begin

                                                                                                                                                  If your organization uses directory synchronization for users, you must select the HdsTrialGroup group object for synchronization to the cloud before you can start a trial for your organization. For instructions, see the Deployment Guide for Cisco Directory Connector.

                                                                                                                                                  1

                                                                                                                                                  Sign in to https://admin.webex.com, and then select Services.

                                                                                                                                                  2

                                                                                                                                                  Under Hybrid Data Security, click Settings.

                                                                                                                                                  3

                                                                                                                                                  In the Service Status section, click Start Trial.

                                                                                                                                                  The service status changes to trial mode.
                                                                                                                                                  4

                                                                                                                                                  Click Add Users and enter the email address of one or more users to pilot using your Hybrid Data Security nodes for encryption and indexing services.

                                                                                                                                                  (If your organization uses directory synchronization, use Active Directory to manage the trial group, HdsTrialGroup.)

                                                                                                                                                  Test Your Hybrid Data Security Deployment

                                                                                                                                                  Use this procedure to test Hybrid Data Security encryption scenarios.

                                                                                                                                                  Before you begin

                                                                                                                                                  • Set up your Hybrid Data Security deployment.

                                                                                                                                                  • Activate the trial, and add several trial users.

                                                                                                                                                  • Ensure that you have access to the syslog to verify that key requests are passing to your Hybrid Data Security deployment.

                                                                                                                                                  1

                                                                                                                                                  Keys for a given space are set by the creator of the space. Sign in to the Webex App as one of the pilot users, and then create a space and invite at least one pilot user and one non-pilot user.

                                                                                                                                                  If you deactivate the Hybrid Data Security deployment, content in spaces that pilot users create is no longer accessible once the client-cached copies of the encryption keys are replaced.

                                                                                                                                                  2

                                                                                                                                                  Send messages to the new space.

                                                                                                                                                  3

                                                                                                                                                  Check the syslog output to verify that the key requests are passing to your Hybrid Data Security deployment.

                                                                                                                                                  1. To check for a user first establishing a secure channel to the KMS, filter on kms.data.method=create and kms.data.type=EPHEMERAL_KEY_COLLECTION:

                                                                                                                                                    You should find an entry such as the following (identifiers shortened for readability):
                                                                                                                                                    2020-07-21 17:35:34.562 (+0000) INFO  KMS [pool-14-thread-1] - [KMS:REQUEST] received, 
                                                                                                                                                    deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/0[~]9 ecdheKid: kms://hds2.org5.portun.us/statickeys/3[~]0 
                                                                                                                                                    (EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_d[~]0, kms.data.method=create, 
                                                                                                                                                    kms.merc.id=8[~]a, kms.merc.sync=false, kms.data.uriHost=hds2.org5.portun.us, kms.data.type=EPHEMERAL_KEY_COLLECTION, 
                                                                                                                                                    kms.data.requestId=9[~]6, kms.data.uri=kms://hds2.org5.portun.us/ecdhe, kms.data.userId=0[~]2
                                                                                                                                                  2. To check for a user requesting an existing key from the KMS, filter on kms.data.method=retrieve and kms.data.type=KEY:

                                                                                                                                                    You should find an entry such as:
                                                                                                                                                    2020-07-21 17:44:19.889 (+0000) INFO  KMS [pool-14-thread-31] - [KMS:REQUEST] received, 
                                                                                                                                                    deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
                                                                                                                                                    (EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_f[~]0, kms.data.method=retrieve, 
                                                                                                                                                    kms.merc.id=c[~]7, kms.merc.sync=false, kms.data.uriHost=ciscospark.com, kms.data.type=KEY, 
                                                                                                                                                    kms.data.requestId=9[~]3, kms.data.uri=kms://ciscospark.com/keys/d[~]2, kms.data.userId=1[~]b
                                                                                                                                                  3. To check for a user requesting the creation of a new KMS key, filter on kms.data.method=create and kms.data.type=KEY_COLLECTION:

                                                                                                                                                    You should find an entry such as:
                                                                                                                                                    2020-07-21 17:44:21.975 (+0000) INFO  KMS [pool-14-thread-33] - [KMS:REQUEST] received, 
                                                                                                                                                    deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
                                                                                                                                                    (EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_4[~]0, kms.data.method=create, 
                                                                                                                                                    kms.merc.id=6[~]e, kms.merc.sync=false, kms.data.uriHost=null, kms.data.type=KEY_COLLECTION, 
                                                                                                                                                    kms.data.requestId=6[~]4, kms.data.uri=/keys, kms.data.userId=1[~]b
                                                                                                                                                  4. To check for a user requesting the creation of a new KMS Resource Object (KRO) when a space or other protected resource is created, filter on kms.data.method=create and kms.data.type=RESOURCE_COLLECTION:

                                                                                                                                                    You should find an entry such as:
                                                                                                                                                    2020-07-21 17:44:22.808 (+0000) INFO  KMS [pool-15-thread-1] - [KMS:REQUEST] received, 
                                                                                                                                                    deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
                                                                                                                                                    (EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_d[~]0, kms.data.method=create, 
                                                                                                                                                    kms.merc.id=5[~]3, kms.merc.sync=true, kms.data.uriHost=null, kms.data.type=RESOURCE_COLLECTION, 
                                                                                                                                                    kms.data.requestId=d[~]e, kms.data.uri=/resources, kms.data.userId=1[~]b

                                                                                                                                                  Monitor Hybrid Data Security Health

                                                                                                                                                  A status indicator within Control Hub shows you whether all is well with the Hybrid Data Security deployment. For more proactive alerting, sign up for email notifications. You'll be notified when there are service-impacting alarms or software upgrades.
                                                                                                                                                  1

                                                                                                                                                  In Control Hub, select Services from the menu on the left side of the screen.

                                                                                                                                                  2

                                                                                                                                                  In the Hybrid Services section, find Hybrid Data Security and click Settings.

                                                                                                                                                  The Hybrid Data Security Settings page appears.
                                                                                                                                                  3

                                                                                                                                                  In the Email Notifications section, type one or more email addresses separated by commas, and press Enter.

                                                                                                                                                  Add or Remove Users from Your Trial

                                                                                                                                                  After you've activated a trial and added the initial set of trial users, you can add or remove trial members at any time while the trial is active.

                                                                                                                                                  If you remove a user from the trial, the user's client will request keys and key creation from the cloud KMS instead of your KMS. If the client needs a key that is stored on your KMS, the cloud KMS will fetch it on the user's behalf.

                                                                                                                                                  If your organization uses directory synchronization, use Active Directory (instead of this procedure) to manage the trial group, HdsTrialGroup; you can view the group members in Control Hub but cannot add or remove them.

                                                                                                                                                  1

                                                                                                                                                  Sign in to Control Hub, and then select Services.

                                                                                                                                                  2

                                                                                                                                                  Under Hybrid Data Security, click Settings.

                                                                                                                                                  3

                                                                                                                                                  In the Trial Mode section of the Service Status area, click Add Users, or click view and edit to remove users from the trial.

                                                                                                                                                  4

                                                                                                                                                  Enter the email address of one or more users to add, or click the X by a user ID to remove the user from the trial. Then click Save.

                                                                                                                                                  Move from Trial to Production

                                                                                                                                                  When you are satisfied that your deployment is working well for the trial users, you can move to production. When you move to production, all users in the organization will use your on-premises Hybrid Data Security domain for encryption keys and other security realm services. You cannot move back to trial mode from production unless you deactivate the service as part of disaster recovery. Reactivating the service requires you to set up a new trial.
                                                                                                                                                  1

                                                                                                                                                  Sign in to Control Hub, and then select Services.

                                                                                                                                                  2

                                                                                                                                                  Under Hybrid Data Security, click Settings.

                                                                                                                                                  3

                                                                                                                                                  In the Service Status section, click Move to Production.

                                                                                                                                                  4

                                                                                                                                                  Confirm that you want to move all of your users to production.

                                                                                                                                                  End Your Trial Without Moving to Production

                                                                                                                                                  If, during your trial, you decide not to go ahead with your Hybrid Data Security deployment, you can deactivate Hybrid Data Security, which ends the trial and moves the trial users back to the cloud data security services. The trial users will lose access to the data that was encrypted during the trial.
                                                                                                                                                  1

                                                                                                                                                  Sign in to Control Hub, and then select Services.

                                                                                                                                                  2

                                                                                                                                                  Under Hybrid Data Security, click Settings.

                                                                                                                                                  3

                                                                                                                                                  In the Deactivate section, click Deactivate.

                                                                                                                                                  4

                                                                                                                                                  Confirm that you want to deactivate the service and end the trial.

                                                                                                                                                  Manage your HDS Deployment

                                                                                                                                                  Manage HDS Deployment

                                                                                                                                                  Use the tasks described here to manage your Hybrid Data Security deployment.

                                                                                                                                                  Set Cluster Upgrade Schedule

                                                                                                                                                  Software upgrades for Hybrid Data Security are done automatically at the cluster level, which ensures that all nodes are always running the same software version. Upgrades are done according to the upgrade schedule for the cluster. When a software upgrade becomes available, you have the option of manually upgrading the cluster before the scheduled upgrade time. You can set a specific upgrade schedule or use the default schedule of 3:00 AM Daily United States: America/Los Angeles. You can also choose to postpone an upcoming upgrade, if necessary.

                                                                                                                                                  To set the upgrade schedule:

                                                                                                                                                  1

                                                                                                                                                  Sign in to Control Hub.

                                                                                                                                                  2

                                                                                                                                                  On the Overview page, under Hybrid Services, select Hybrid Data Security.

                                                                                                                                                  3

                                                                                                                                                  On the Hybrid Data Security Resources page, select the cluster.

                                                                                                                                                  4

                                                                                                                                                  In the Overview panel on the right, under Cluster Settings, select the cluster name.

                                                                                                                                                  5

                                                                                                                                                  On the Settings page, under Upgrade, select the time and time zone for the upgrade schedule.

                                                                                                                                                  Note: Under the time zone, the next available upgrade date and time is displayed. You can postpone the upgrade to the following day, if needed, by clicking Postpone.

                                                                                                                                                  Change the Node Configuration

                                                                                                                                                  Occasionally you may need to change the configuration of your Hybrid Data Security node for a reason such as:
                                                                                                                                                  • Changing x.509 certificates due to expiration or other reasons.

                                                                                                                                                    We don't support changing the CN domain name of a certificate. The domain must match the original domain used to register the cluster.

                                                                                                                                                  • Updating database settings to change to a replica of the PostgreSQL or Microsoft SQL Server database.

                                                                                                                                                    We don’t support migrating data from PostgreSQL to Microsoft SQL Server, or the opposite way. To switch the database environment, start a new deployment of Hybrid Data Security.

                                                                                                                                                  • Creating a new configuration to prepare a new data center.

                                                                                                                                                  Also, for security purposes, Hybrid Data Security uses service account passwords that have a nine-month lifespan. After the HDS Setup tool generates these passwords, you deploy them to each of your HDS nodes in the ISO config file. When your organization's passwords are nearing expiration, you receive a notice from the Webex team to reset the password for your machine account. (The email includes the text, "Use the machine account API to update the password.") If your passwords haven't expired yet, the tool gives you two options:

                                                                                                                                                  • Soft reset—The old and new passwords both work for up to 10 days. Use this period to replace the ISO file on the nodes gradually.

                                                                                                                                                  • Hard reset—The old passwords stop working immediately.

                                                                                                                                                  If your passwords expire without a reset, it impacts your HDS service, requiring an immediate hard reset and replacement of the ISO file on all nodes.

                                                                                                                                                  Use this procedure to generate a new configuration ISO file and apply it to your cluster.

                                                                                                                                                  Before you begin

                                                                                                                                                  • The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Control Hub account with full administrator rights for your organization.

                                                                                                                                                    If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in 1.e. This table gives some possible environment variables:

                                                                                                                                                    Description

                                                                                                                                                    Variable

                                                                                                                                                    HTTP Proxy without authentication

                                                                                                                                                    GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT

                                                                                                                                                    HTTPS Proxy without authentication

                                                                                                                                                    GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT

                                                                                                                                                    HTTP Proxy with authentication

                                                                                                                                                    GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT

                                                                                                                                                    HTTPS Proxy with authentication

                                                                                                                                                    GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT

                                                                                                                                                  • You need a copy of the current configuration ISO file to generate a new configuration. The ISO contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the ISO when you make configuration changes, including database credentials, certificate updates, or changes to authorization policy.

                                                                                                                                                  1

                                                                                                                                                  Using Docker on a local machine, run the HDS Setup Tool.

                                                                                                                                                  1. At your machine's command line, enter the appropriate command for your environment:

                                                                                                                                                    In regular environments:

                                                                                                                                                    docker rmi ciscocitg/hds-setup:stable

                                                                                                                                                    In FedRAMP environments:

                                                                                                                                                    docker rmi ciscocitg/hds-setup-fedramp:stable

                                                                                                                                                    This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.

                                                                                                                                                  2. To sign in to the Docker image registry, enter the following:

                                                                                                                                                    docker login -u hdscustomersro
                                                                                                                                                  3. At the password prompt, enter this hash:

                                                                                                                                                    dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo
                                                                                                                                                  4. Download the latest stable image for your environment:

                                                                                                                                                    In regular environments:

                                                                                                                                                    docker pull ciscocitg/hds-setup:stable

                                                                                                                                                    In FedRAMP environments:

                                                                                                                                                    docker pull ciscocitg/hds-setup-fedramp:stable

                                                                                                                                                    Make sure you pull the latest Setup tool for this procedure. Versions of the tool created before February 22, 2018 don’t have the password reset screens.

                                                                                                                                                  5. When the pull completes, enter the appropriate command for your environment:

                                                                                                                                                    • In regular environments without a proxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable
                                                                                                                                                    • In regular environments with an HTTP proxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable
                                                                                                                                                    • In regular environments with an HTTPSproxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable
                                                                                                                                                    • In FedRAMP environments without a proxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable
                                                                                                                                                    • In FedRAMP environments with an HTTP proxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable
                                                                                                                                                    • In FedRAMP environments with an HTTPS proxy:

                                                                                                                                                      docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable

                                                                                                                                                    When the container is running, you see "Express server listening on port 8080."

                                                                                                                                                  6. Use a browser to connect to the localhost, http://127.0.0.1:8080.

                                                                                                                                                    The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost.

                                                                                                                                                  7. When prompted, enter your Control Hub customer sign-in credentials and then click Accept to continue.

                                                                                                                                                  8. Import the current configuration ISO file.

                                                                                                                                                  9. Follow the prompts to complete the tool and download the updated file.

                                                                                                                                                    To shut down the Setup tool, type CTRL+C.

                                                                                                                                                  10. Create a backup copy of the updated file in another data center.

                                                                                                                                                  2

                                                                                                                                                  If you only have one HDS node running, create a new Hybrid Data Security node VM and register it using the new configuration ISO file. For more detailed instructions, see Create and Register More Nodes.

                                                                                                                                                  1. Install the HDS host OVA.

                                                                                                                                                  2. Set up the HDS VM.

                                                                                                                                                  3. Mount the updated configuration file.

                                                                                                                                                  4. Register the new node in Control Hub.

                                                                                                                                                  3

                                                                                                                                                  For existing HDS nodes that are running the older configuration file, mount the ISO file. Perform the following procedure on each node in turn, updating each node before turning off the next node:

                                                                                                                                                  1. Turn off the virtual machine.

                                                                                                                                                  2. In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.

                                                                                                                                                  3. Click CD/DVD Drive 1, select the option to mount from an ISO file, and browse to the location where you downloaded the new configuration ISO file.

                                                                                                                                                  4. Check Connect at power on.

                                                                                                                                                  5. Save your changes and power on the virtual machine.

                                                                                                                                                  4

                                                                                                                                                  Repeat step 3 to replace the configuration on each remaining node that is running the old configuration.

                                                                                                                                                  Turn off Blocked External DNS Resolution Mode

                                                                                                                                                  When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. If the node's DNS server can't resolve public DNS names, the node automatically goes into Blocked External DNS Resolution mode.

                                                                                                                                                  If your nodes are able to resolve public DNS names through internal DNS servers, you can turn off this mode by rerunning the proxy connection test on each node.

                                                                                                                                                  Before you begin

                                                                                                                                                  Ensure that your internal DNS servers can resolve public DNS names, and that your nodes can communicate with them.
                                                                                                                                                  1

                                                                                                                                                  In a web browser, open the Hybrid Data Security node interface (IP address/setup, for example, https://192.0.2.0/setup), enter the admin credentials you set up for the node, and then click Sign In.

                                                                                                                                                  2

                                                                                                                                                  Go to Overview (the default page).

                                                                                                                                                  When enabled, Blocked External DNS Resolution is set to Yes.

                                                                                                                                                  3

                                                                                                                                                  Go to the Trust Store & Proxy page.

                                                                                                                                                  4

                                                                                                                                                  Click Check Proxy Connection.

                                                                                                                                                  If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server and will remain in this mode. Otherwise, after you reboot the node and go back to the Overview page, Blocked External DNS Resolution should be set to no.

                                                                                                                                                  What to do next

                                                                                                                                                  Repeat the proxy connection test on each node in your Hybrid Data Security cluster.

                                                                                                                                                  Remove a Node

                                                                                                                                                  Use this procedure to remove a Hybrid Data Security node from the Webex cloud. After you remove the node from the cluster, delete the virtual machine to prevent further access to your security data.
                                                                                                                                                  1

                                                                                                                                                  Use the VMware vSphere client on your computer to log into the ESXi virtual host and power off the virtual machine.

                                                                                                                                                  2

                                                                                                                                                  Remove the node:

                                                                                                                                                  1. Sign in to Control Hub, and then select Services.

                                                                                                                                                  2. On the Hybrid Data Security card, click View All to display the Hybrid Data Security Resources page.

                                                                                                                                                  3. Select your cluster to display its Overview panel.

                                                                                                                                                  4. Click Open nodes list.

                                                                                                                                                  5. On the Nodes tab, select the node you want to remove.

                                                                                                                                                  6. Click Actions > Deregister node.

                                                                                                                                                  3

                                                                                                                                                  In the vSphere client, delete the VM. (In the left navigation pane, right-click on the VM and click Delete.)

                                                                                                                                                  If you don’t delete the VM, remember to unmount the configuration ISO file. Without the ISO file, you can't use the VM to access your security data.

                                                                                                                                                  Disaster Recovery using Standby Data Center

                                                                                                                                                  The most critical service that your Hybrid Data Security cluster provides is the creation and storage of keys used to encrypt messages and other content stored in the Webex cloud. For each user within the organization who is assigned to Hybrid Data Security, new key creation requests are routed to the cluster. The cluster is also responsible for returning the keys that it's created to any users authorized to retrieve them, for example, members of a conversation space.

                                                                                                                                                  Because the cluster performs the critical function of providing these keys, it's imperative that the cluster remains running and that proper backups are maintained. Loss of the Hybrid Data Security database or of the configuration ISO used for the schema will result in UNRECOVERABLE LOSS of customer content. The following practices are mandatory to prevent such a loss:

                                                                                                                                                  If a disaster causes the HDS deployment in the primary data center to become unavailable, follow this procedure to manually failover to the standby data center.

                                                                                                                                                  1

                                                                                                                                                  Start the HDS Setup tool and follow the steps mentioned in Create a Configuration ISO for the HDS Hosts.

                                                                                                                                                  2

                                                                                                                                                  After configuring the Syslogd server, click on Advanced Settings

                                                                                                                                                  3

                                                                                                                                                  On the Advanced Settings page, add the configuration below or remove the passiveMode configuration to make the node active. The node can handle traffic once this is configured.

                                                                                                                                                  
                                                                                                                                                  passiveMode: 'false'
                                                                                                                                                  

                                                                                                                                                  4

                                                                                                                                                  Complete the configuration process and save the ISO file in a location that's easy to find.

                                                                                                                                                  5

                                                                                                                                                  Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes.

                                                                                                                                                  6

                                                                                                                                                  In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings..

                                                                                                                                                  7

                                                                                                                                                  Click Edit Settings >CD/DVD Drive 1 and select Datastore ISO File.

                                                                                                                                                  Make sure Connected and Connect at power on are checked so that updated configuration changes can take effect after starting the nodes.

                                                                                                                                                  8

                                                                                                                                                  Power on the HDS node and make sure there are no alarms for at least 15 minutes.

                                                                                                                                                  9

                                                                                                                                                  Repeat the process for every node in the standby data center.

                                                                                                                                                  Check the syslog output to verify that the nodes of the standby data center are not in passive mode. “KMS configured in passive mode” should not appear in the syslogs.

                                                                                                                                                  What to do next

                                                                                                                                                  After failover, if the primary data center becomes active again, place the standby data center in passive mode again by following the steps described in Setup Standby Data Center for Disaster Recovery.

                                                                                                                                                  (Optional) Unmount ISO After HDS Configuration

                                                                                                                                                  The standard HDS configuration runs with the ISO mounted. But, some customers prefer not leaving ISO files continuously mounted. You can unmount the ISO file after all HDS nodes pick up the new configuration.

                                                                                                                                                  You still use the ISO files to make configuration changes. When you create a new ISO or update an ISO through the Setup Tool, you must mount the updated ISO on all your HDS nodes. Once all your nodes have picked up the configuration changes, you can unmount the ISO again with this procedure.

                                                                                                                                                  Before you begin

                                                                                                                                                  Upgrade all your HDS nodes to version 2021.01.22.4720 or later.

                                                                                                                                                  1

                                                                                                                                                  Shut down one of your HDS nodes.

                                                                                                                                                  2

                                                                                                                                                  In the vCenter Server Appliance, select the HDS node.

                                                                                                                                                  3

                                                                                                                                                  Choose Edit Settings > CD/DVD drive and uncheck Datastore ISO File.

                                                                                                                                                  4

                                                                                                                                                  Power on the HDS node and ensure there are no alarms for atleast 20 minutes.

                                                                                                                                                  5

                                                                                                                                                  Repeat for each HDS node in turn.

                                                                                                                                                  Troubleshoot Hybrid Data Security

                                                                                                                                                  View Alerts and Troubleshoot

                                                                                                                                                  A Hybrid Data Security deployment is considered unavailable if all nodes in the cluster are unreachable, or the cluster is working so slowly that requests time out. If users cannot reach your Hybrid Data Security cluster, they experience the following symptoms:

                                                                                                                                                  • New spaces cannot be created (unable to create new keys)

                                                                                                                                                  • Messages and space titles fail to decrypt for:

                                                                                                                                                    • New users added to a space (unable to fetch keys)

                                                                                                                                                    • Existing users in a space using a new client (unable to fetch keys)

                                                                                                                                                  • Existing users in a space will continue to run successfully as long as their clients have a cache of the encryption keys

                                                                                                                                                  It's important that you properly monitor your Hybrid Data Security cluster and address any alerts promptly to avoid disruption of service.

                                                                                                                                                  Alerts

                                                                                                                                                  If there is a problem with the Hybrid Data Security setup, Control Hub displays alerts to the organization administrator, and sends emails to the configured email address. The alerts cover many common scenarios.

                                                                                                                                                  Table 1. Common Issues and the Steps to Resolve Them

                                                                                                                                                  Alert

                                                                                                                                                  Action

                                                                                                                                                  Local database access failure.

                                                                                                                                                  Check for database errors or local network issues.

                                                                                                                                                  Local database connection failure.

                                                                                                                                                  Check that the database server is available, and the right service account credentials were used in node configuration.

                                                                                                                                                  Cloud service access failure.

                                                                                                                                                  Check that the nodes can access the Webex servers as specified in External connectivity requirements.

                                                                                                                                                  Renewing cloud service registration.

                                                                                                                                                  Registration to cloud services was dropped. Renewal of registration is in progress.

                                                                                                                                                  Cloud service registration dropped.

                                                                                                                                                  Registration to cloud services terminated. Service is shutting down.

                                                                                                                                                  Service not yet activated.

                                                                                                                                                  Activate a trial, or finish moving the trial to production.

                                                                                                                                                  Configured domain does not match server certificate.

                                                                                                                                                  Ensure that your server certificate matches the configured service activation domain.

                                                                                                                                                  The most likely cause is that the certificate CN was recently changed and is now different from the CN that was used during initial setup.

                                                                                                                                                  Failed to authenticate to cloud services.

                                                                                                                                                  Check for accuracy and possible expiration of service account credentials.

                                                                                                                                                  Failed to open local keystore file.

                                                                                                                                                  Check for integrity and password accuracy on local keystore file.

                                                                                                                                                  Local server certificate is invalid.

                                                                                                                                                  Check the server certificate's expiration date and confirm that it was issued by a trusted Certificate Authority.

                                                                                                                                                  Unable to post metrics.

                                                                                                                                                  Check local network access to external cloud services.

                                                                                                                                                  /media/configdrive/hds directory does not exist.

                                                                                                                                                  Check the ISO mount configuration on virtual host. Verify that the ISO file exists, that it is configured to mount on reboot, and that it mounts successfully.

                                                                                                                                                  Troubleshoot Hybrid Data Security

                                                                                                                                                  Use the following general guidelines when troubleshooting problems with Hybrid Data Security.
                                                                                                                                                  1

                                                                                                                                                  Review Control Hub for any alerts and fix any items you find there.

                                                                                                                                                  2

                                                                                                                                                  Review the syslog server output for activity from the Hybrid Data Security deployment.

                                                                                                                                                  3

                                                                                                                                                  Contact Cisco support.

                                                                                                                                                  Other Notes

                                                                                                                                                  Known Issues for Hybrid Data Security

                                                                                                                                                  • If you shut down your Hybrid Data Security cluster (by deleting it in Control Hub or by shutting down all nodes), lose your configuration ISO file, or lose access to the keystore database, your Webex App users can no longer use spaces under their People list that were created with keys from your KMS. This applies to both trial and production deployments. We do not currently have a workaround or fix for this issue and urge you not to shut down your HDS services once they are handling active user accounts.

                                                                                                                                                  • A client which has an existing ECDH connection to a KMS maintains that connection for a period of time (likely one hour). When a user becomes a member of a Hybrid Data Security trial, the user's client continues to use the existing ECDH connection until it times out. Alternatively, the user can sign out and back in to the Webex App app to update the location that the app contacts for encryption keys.

                                                                                                                                                    The same behavior occurs when you move a trial to production for the organization. All non-trial users with existing ECDH connections to the previous data security services will continue to use those services until the ECDH connection is renegotiated (through timeout or by signing out and back in).

                                                                                                                                                  Use OpenSSL to Generate a PKCS12 File

                                                                                                                                                  Before you begin

                                                                                                                                                  • OpenSSL is one tool that can be used to make the PKCS12 file in the proper format for loading in the HDS Setup Tool. There are other ways to do this, and we do not support or promote one way over another.

                                                                                                                                                  • If you do choose to use OpenSSL, we are providing this procedure as a guideline to help you create a file that meets the X.509 certificate requirements in X.509 Certificate Requirements. Understand those requirements before you continue.

                                                                                                                                                  • Install OpenSSL in a supported environment. See https://www.openssl.org for the software and documentation.

                                                                                                                                                  • Create a private key.

                                                                                                                                                  • Start this procedure when you receive the server certificate from your Certificate Authority (CA).

                                                                                                                                                  1

                                                                                                                                                  When you receive the server certificate from your CA, save it as hdsnode.pem.

                                                                                                                                                  2

                                                                                                                                                  Display the certificate as text, and verify the details.

                                                                                                                                                  openssl x509 -text -noout -in hdsnode.pem

                                                                                                                                                  3

                                                                                                                                                  Use a text editor to create a certificate bundle file called hdsnode-bundle.pem. The bundle file must include the server certificate, any intermediate CA certificates, and the root CA certificates, in the format below:

                                                                                                                                                  -----BEGIN CERTIFICATE-----
                                                                                                                                                  ### Server certificate. ###
                                                                                                                                                  -----END CERTIFICATE-----
                                                                                                                                                  -----BEGIN CERTIFICATE-----
                                                                                                                                                  ###  Intermediate CA certificate. ###
                                                                                                                                                  -----END CERTIFICATE-----
                                                                                                                                                  -----BEGIN CERTIFICATE-----
                                                                                                                                                  ###  Root CA certificate. ###
                                                                                                                                                  -----END CERTIFICATE-----

                                                                                                                                                  4

                                                                                                                                                  Create the .p12 file with the friendly name kms-private-key.

                                                                                                                                                  openssl pkcs12 -export -inkey hdsnode.key -in hdsnode-bundle.pem -name kms-private-key -caname kms-private-key -out hdsnode.p12

                                                                                                                                                  5

                                                                                                                                                  Check the server certificate details.

                                                                                                                                                  1. openssl pkcs12 -in hdsnode.p12

                                                                                                                                                  2. Enter a password at the prompt to encrypt the private key so that it is listed in the output. Then, verify that the private key and the first certificate include the lines friendlyName: kms-private-key.

                                                                                                                                                    Example:

                                                                                                                                                    bash$ openssl pkcs12 -in hdsnode.p12
                                                                                                                                                    Enter Import Password:
                                                                                                                                                    MAC verified OK
                                                                                                                                                    Bag Attributes
                                                                                                                                                        friendlyName: kms-private-key
                                                                                                                                                        localKeyID: 54 69 6D 65 20 31 34 39 30 37 33 32 35 30 39 33 31 34 
                                                                                                                                                    Key Attributes: <No Attributes>
                                                                                                                                                    Enter PEM pass phrase:
                                                                                                                                                    Verifying - Enter PEM pass phrase:
                                                                                                                                                    -----BEGIN ENCRYPTED PRIVATE KEY-----
                                                                                                                                                    <redacted>
                                                                                                                                                    -----END ENCRYPTED PRIVATE KEY-----
                                                                                                                                                    Bag Attributes
                                                                                                                                                        friendlyName: kms-private-key
                                                                                                                                                        localKeyID: 54 69 6D 65 20 31 34 39 30 37 33 32 35 30 39 33 31 34 
                                                                                                                                                    subject=/CN=hds1.org6.portun.us
                                                                                                                                                    issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
                                                                                                                                                    -----BEGIN CERTIFICATE-----
                                                                                                                                                    <redacted>
                                                                                                                                                    -----END CERTIFICATE-----
                                                                                                                                                    Bag Attributes
                                                                                                                                                        friendlyName: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
                                                                                                                                                    subject=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
                                                                                                                                                    issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3
                                                                                                                                                    -----BEGIN CERTIFICATE-----
                                                                                                                                                    <redacted>
                                                                                                                                                    -----END CERTIFICATE-----

                                                                                                                                                  What to do next

                                                                                                                                                  Return to Complete the Prerequisites for Hybrid Data Security. You will use the hdsnode.p12 file, and the password you've set for it, in Create a Configuration ISO for the HDS Hosts.

                                                                                                                                                  You can reuse these files to request a new certificate when the original certificate expires.

                                                                                                                                                  Traffic between the HDS Nodes and the Cloud

                                                                                                                                                  Outbound Metrics Collection Traffic

                                                                                                                                                  The Hybrid Data Security nodes send certain metrics to the Webex cloud. These include system metrics for heap max, heap used, CPU load, and thread count; metrics on synchronous and asynchronous threads; metrics on alerts involving a threshold of encryption connections, latency, or a request queue length; metrics on the datastore; and encryption connection metrics. The nodes send encrypted key material over an out-of-band (separate from the request) channel.

                                                                                                                                                  Inbound Traffic

                                                                                                                                                  The Hybrid Data Security nodes receive the following types of inbound traffic from the Webex cloud:

                                                                                                                                                  • Encryption requests from clients, which are routed by the encryption service

                                                                                                                                                  • Upgrades to the node software

                                                                                                                                                  Configure Squid Proxies for Hybrid Data Security

                                                                                                                                                  Websocket Cannot Connect Through Squid Proxy

                                                                                                                                                  Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:) connections that Hybrid Data Security requires. These sections give guidance on how to configure various versions of Squid to ignore wss: traffic for proper operation of the services.

                                                                                                                                                  Squid 4 and 5

                                                                                                                                                  Add the on_unsupported_protocol directive to squid.conf:

                                                                                                                                                  on_unsupported_protocol tunnel all

                                                                                                                                                  Squid 3.5.27

                                                                                                                                                  We successfully tested Hybrid Data Security with the following rules added to squid.conf. These rules are subject to change as we develop features and update the Webex cloud.

                                                                                                                                                  acl wssMercuryConnection ssl::server_name_regex mercury-connection
                                                                                                                                                  
                                                                                                                                                  ssl_bump splice wssMercuryConnection
                                                                                                                                                  
                                                                                                                                                  acl step1 at_step SslBump1
                                                                                                                                                  acl step2 at_step SslBump2
                                                                                                                                                  acl step3 at_step SslBump3
                                                                                                                                                  ssl_bump peek step1 all
                                                                                                                                                  ssl_bump stare step2 all
                                                                                                                                                  ssl_bump bump step3 all
                                                                                                                                                  Was this article helpful?