- Home
- /
- Article
Deployment guide for Webex Hybrid Data Security
Preface
New and changed information
Date |
Changes Made |
---|---|
October 20, 2023 |
|
August 07, 2023 |
|
May 23, 2023 |
|
December 06, 2022 |
|
November 23, 2022 |
|
October 13, 2021 |
Docker Desktop needs to run a setup program before you can install HDS nodes. See Docker Desktop Requirements. |
June 24, 2021 |
Noted that you can reuse the private key file and CSR to request another certificate. See Use OpenSSL to Generate a PKCS12 File for details. |
April 30, 2021 |
Changed the VM requirement for local hard disk space to 30 GB. See Virtual Host Requirements for details. |
February 24, 2021 |
HDS Setup Tool can now run behind a proxy. See Create a Configuration ISO for the HDS Hosts for details. |
February 2, 2021 |
HDS can now run without a mounted ISO file. See (Optional) Unmount ISO After HDS Configuration for details. |
January 11, 2021 |
Added info on HDS Setup tool and proxies to Create a Configuration ISO for the HDS Hosts. |
October 13, 2020 |
Updated Download Installation Files. |
October 8, 2020 |
Updated Create a Configuration ISO for the HDS Hosts and Change the Node Configuration with commands for FedRAMP environments. |
August 14, 2020 |
Updated Create a Configuration ISO for the HDS Hosts and Change the Node Configuration with changes to the sign-in process. |
August 5, 2020 |
Updated Test Your Hybrid Data Security Deployment for changes in log messages. Updated Virtual Host Requirements to remove maximum number of hosts. |
June 16, 2020 |
Updated Remove a Node for changes in the Control Hub UI. |
June 4, 2020 |
Updated Create a Configuration ISO for the HDS Hosts for changes in the Advanced Settings that you might set. |
May 29, 2020 |
Updated Create a Configuration ISO for the HDS Hosts to show you can also use TLS with SQL Server databases, UI changes, and other clarifications. |
May 5, 2020 |
Updated Virtual Host Requirements to show new requirement of ESXi 6.5. |
April 21, 2020 |
Updated External connectivity requirements with new Americas CI hosts. |
April 1, 2020 |
Updated External connectivity requirements with information on regional CI hosts. |
February 20, 2020 | Updated Create a Configuration ISO for the HDS Hosts with information on new optional Advanced Settings screen in the HDS Setup Tool. |
February 4, 2020 | Updated Proxy Server Requirements. |
December 16, 2019 | Clarified the requirement for Blocked External DNS Resolution Mode to work in Proxy Server Requirements. |
November 19, 2019 |
Added information about Blocked External DNS Resolution Mode in the following sections: |
November 8, 2019 |
You can now configure network settings for a node while deploying the OVA rather than afterwards. Updated the following sections accordingly: The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions. |
September 6, 2019 |
Added SQL Server Standard to Database server requirements. |
August 29, 2019 | Added Configure Squid Proxies for Hybrid Data Security appendix with guidance on configuring Squid proxies to ignore websocket traffic for proper operation. |
August 20, 2019 |
Added and updated sections to cover proxy support for Hybrid Data Security node communications to the Webex cloud. To access just the proxy support content for an existing deployment, see the Proxy Support for Hybrid Data Security and Webex Video Mesh help article. |
June 13, 2019 | Updated Trial to Production Task Flow with a reminder to synchronize the HdsTrialGroup group object before starting a trial if your organization uses directory synchronization. |
March 6, 2019 |
|
February 28, 2019 |
|
February 26, 2019 |
|
January 24, 2019 |
|
November 5, 2018 |
|
October 19, 2018 |
|
July 31, 2018 |
|
May 21, 2018 |
Changed terminology to reflect the rebranding of Cisco Spark:
|
April 11, 2018 |
|
February 22, 2018 |
|
February 15, 2018 |
|
January 18, 2018 |
|
November 2, 2017 |
|
August 18, 2017 |
First published |
Get Started with Hybrid Data Security
Hybrid Data Security Overview
From day one, data security has been the primary focus in designing Webex App. The cornerstone of this security is end-to-end content encryption, enabled by Webex App clients interacting with the Key Management Service (KMS). The KMS is responsible for creating and managing the cryptographic keys that clients use to dynamically encrypt and decrypt messages and files.
By default, all Webex App customers get end-to-end encryption with dynamic keys stored in the cloud KMS, in Cisco's security realm. Hybrid Data Security moves the KMS and other security-related functions to your enterprise data center, so nobody but you holds the keys to your encrypted content.
Security Realm Architecture
The Webex cloud architecture separates different types of service into separate realms, or trust domains, as depicted below.
To further understand Hybrid Data Security, let's first look at this pure cloud case, where Cisco is providing all functions in its cloud realms. The identity service, the only place where users can be directly correlated with their personal information such as email address, is logically and physically separate from the security realm in data center B. Both are in turn separate from the realm where encrypted content is ultimately stored, in data center C.
In this diagram, the client is the Webex App running on a user's laptop, and has authenticated with the identity service. When the user composes a message to send to a space, the following steps take place:
-
The client establishes a secure connection with the key management service (KMS), then requests a key to encrypt the message. The secure connection uses ECDH, and the KMS encrypts the key using an AES-256 master key.
-
The message is encrypted before it leaves the client. The client sends it to the indexing service, which creates encrypted search indexes to aid in future searches for the content.
-
The encrypted message is sent to the compliance service for compliance checks.
-
The encrypted message is stored in the storage realm.
When you deploy Hybrid Data Security, you move the security realm functions (KMS, indexing, and compliance) to your on-premises data center. The other cloud services that make up Webex (including identity and content storage) remain in Cisco’s realms.
Collaborating with Other Organizations
Users in your organization may regularly use Webex App to collaborate with external participants in other organizations. When one of your users requests a key for a space that is owned by your organization (because it was created by one of your users) your KMS sends the key to the client over an ECDH secured channel. However, when another organization owns the key for the space, your KMS routes the request out to the Webex cloud through a separate ECDH channel to get the key from the appropriate KMS, and then returns the key to your user on the original channel.
The KMS service running on Org A validates the connections to KMSs in other organizations using x.509 PKI certificates. See Prepare Your Environment for details on generating an x.509 certificate to use with your Hybrid Data Security deployment.
Expectations for Deploying Hybrid Data Security
A Hybrid Data Security deployment requires significant customer commitment and an awareness of the risks that come with owning encryption keys.
To deploy Hybrid Data Security, you must provide:
-
A secure data center in a country that is a supported location for the Cisco Webex Teams plans.
Complete loss of either the configuration ISO that you build for Hybrid Data Security or the database that you provide will result in the loss of the keys. Key loss prevents users from decrypting space content and other encrypted data in Webex App. If this happens, you can build a new deployment, but only new content will be visible. To avoid loss of access to data, you must:
-
Manage the backup and recovery of the database and the configuration ISO.
-
Be prepared to perform quick disaster recovery if a catastrophe occurs, such as database disk failure or data center disaster.
There is no mechanism to move keys back to the Cloud after an HDS deployment.
High-level Setup Process
This document covers the setup and management of a Hybrid Data Security deployment:
Set up Hybrid Data Security—This includes preparing required infrastructure and installing Hybrid Data Security software, testing your deployment with a subset of users in trial mode, and, once your testing is complete, moving to production. This converts the entire organization to use your Hybrid Data Security cluster for security functions.
The setup, trial, and production phases are covered in detail in the next three chapters.
-
Maintain your Hybrid Data Security deployment—The Webex cloud automatically provides ongoing upgrades. Your IT department can provide tier one support for this deployment, and engage Cisco support as needed. You can use on-screen notifications and set up email-based alerts in Control Hub.
-
Understand common alerts, troubleshooting steps, and known issues—If you run into trouble deploying or using Hybrid Data Security, the last chapter of this guide and the Known Issues appendix may help you determine and fix the issue.
Hybrid Data Security Deployment Model
Within your enterprise data center, you deploy Hybrid Data Security as a single cluster of nodes on separate virtual hosts. The nodes communicate with the Webex cloud through secure websockets and secure HTTP.
During the installation process, we provide you with the OVA file to set up the virtual appliance on the VMs that you provide. You use the HDS Setup Tool to create a custom cluster configuration ISO file that you mount on each node. The Hybrid Data Security cluster uses your provided Syslogd server and PostgreSQL or Microsoft SQL Server database. (You configure the Syslogd and database connection details in the HDS Setup Tool.)
The minimum number of nodes you can have in a cluster is two. We recommend at least three per cluster. Having multiple nodes ensures that service is not interrupted during a software upgrade or other maintenance activity on a node. (The Webex cloud only upgrades one node at a time.)
All nodes in a cluster access the same key datastore, and log activity to the same syslog server. The nodes themselves are stateless, and handle key requests in round-robin fashion, as directed by the cloud.
Nodes become active when you register them in Control Hub. To take an individual node out of service, you can deregister it, and later reregister it if needed.
We support only a single cluster per organization.
Hybrid Data Security Trial Mode
After setting up a Hybrid Data Security deployment, you first try it with a set of pilot users. During the trial period, these users use your on-premises Hybrid Data Security domain for encryption keys and other security realm services. Your other users continue to use the cloud security realm.
If you decide not to continue with the deployment during the trial and deactivate the service, the pilot users and any users they have interacted with by creating new spaces during the trial period will lose access to the messages and content. They will see “This message cannot be decrypted” in the Webex App.
If you are satisfied that your deployment is working well for the trial users and you are ready to extend Hybrid Data Security to all of your users, you move the deployment to production. Pilot users continue to have access to the keys that were in use during the trial. However, you cannot move back and forth between production mode and the original trial. If you must deactivate the service, such as to perform disaster recovery, when you reactivate you must start a new trial and set up the set of pilot users for the new trial before moving back to production mode. Whether users retain access to data at this point depends on whether you have successfully maintained backups of the key data store and the ISO configuration file for the Hybrid Data Security nodes in your cluster.
Standby Data Center for Disaster Recovery
During deployment, you set up a secure standby data center. In the event of a data center disaster, you can manually fail your deployment over to the standby data center.
The databases of the active and standby data centers are in sync with each other which will minimize the time taken to perform the failover. The ISO file of the standby data center is updated with additional configurations which ensure that the nodes are registered to the organization, but will not handle traffic. Hence, the nodes of the standby data center always remain up-to-date with the latest version of HDS software.
The active Hybrid Data Security nodes must always be in the same data center as the active database server.
Setup Standby Data Center for Disaster Recovery
Follow the steps below to configure the ISO file of the standby data center:
Before you begin
-
The standby data center should mirror the production environment of VMs and a backup PostgreSQL or Microsoft SQL Server database. For example, if production has 3 VMs running HDS nodes, the backup environment should have 3 VMs. (See Standby Data Center for Disaster Recovery for an overview of this failover model.)
-
Make sure database sync is enabled between the database of active and passive cluster nodes.
1 |
Start the HDS Setup tool and follow the steps mentioned in Create a Configuration ISO for the HDS Hosts. The ISO file must be a copy of the original ISO file of the primary data center onto which the following configuration updates are to be made. |
2 |
After configuring the Syslogd server, click on Advanced Settings |
3 |
On the Advanced Settings page, add the configuration below to place the node in passive mode. In this mode the node will be registered to the organization and connected to cloud, but will not handle any traffic.
|
4 |
Complete the configuration process and save the ISO file in a location that's easy to find. |
5 |
Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes. |
6 |
In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.. |
7 |
Click Edit Settings >CD/DVD Drive 1 and select Datastore ISO File. Make sure Connected and Connect at power on are checked so that updated configuration changes can take effect after starting the nodes. |
8 |
Power on the HDS node and make sure there are no alarms for at least 15 minutes. |
9 |
Repeat the process for every node in the standby data center. Check the syslogs to verify that the nodes are in passive mode. You should be able to view the message “KMS configured in passive mode” in the syslogs. |
What to do next
After configuring passiveMode
in the ISO file and saving it, you can create another copy of the ISO file without the passiveMode
configuration and save it in a secure location. This copy of the ISO file without passiveMode
configured can help in a quick failover process during disaster recovery. See Disaster Recovery using Standby Data Center for the detailed failover procedure.
Proxy Support
Hybrid Data Security supports explicit, transparent inspecting, and non-inspecting proxies. You can tie these proxies to your deployment so that you can secure and monitor traffic from the enterprise out to the cloud. You can use a platform admin interface on the nodes for certificate management and to check the overall connectivity status after you set up the proxy on the nodes.
The Hybrid Data Security nodes support the following proxy options:
-
No proxy—The default if you do not use the HDS node setup Trust Store & Proxy configuration to integrate a proxy. No certificate update is required.
-
Transparent non-inspecting proxy—The nodes are not configured to use a specific proxy server address and should not require any changes to work with a non-inspecting proxy. No certificate update is required.
-
Transparent tunneling or inspecting proxy—The nodes are not configured to use a specific proxy server address. No HTTP or HTTPS configuration changes are necessary on the nodes. However, the nodes need a root certificate so that they trust the proxy. Inspecting proxies are typically used by IT to enforce policies on which websites can be visited and which types of content are not permitted. This type of proxy decrypts all your traffic (even HTTPS).
-
Explicit proxy—With explicit proxy, you tell the HDS nodes which proxy server and authentication scheme to use. To configure an explicit proxy, you must enter the following information on each node:
-
Proxy IP/FQDN—Address that can be used to reach the proxy machine.
-
Proxy Port—A port number that the proxy uses to listen for proxied traffic.
-
Proxy Protocol—Depending on what your proxy server supports, choose between the following protocols:
-
HTTP—Views and controls all requests that the client sends.
-
HTTPS—Provides a channel to the server. The client receives and validates the server's certificate.
-
-
Authentication Type—Choose from among the following authentication types:
-
None—No further authentication is required.
Available if you select either HTTP or HTTPS as the proxy protocol.
-
Basic—Used for an HTTP User Agent to provide a user name and password when making a request. Uses Base64 encoding.
Available if you select either HTTP or HTTPS as the proxy protocol.
Requires you to enter the user name and password on each node.
-
Digest—Used to confirm the account before sending sensitive information. Applies a hash function on the user name and password before sending over the network.
Available only if you select HTTPS as the proxy protocol.
Requires you to enter the user name and password on each node.
-
-
Example of Hybrid Data Security Nodes and Proxy
This diagram shows an example connection between the Hybrid Data Security, network and a proxy. For the transparent inspecting and HTTPS explicit inspecting proxy options, the same root certificate must be installed on the proxy and on the Hybrid Data Security nodes.
Blocked External DNS Resolution Mode (Explicit Proxy Configurations)
When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. In deployments with explicit proxy configurations that do not allow external DNS resolution for internal clients, if the node can't query the DNS servers, it automatically goes into Blocked External DNS Resolution mode. In this mode, node registration and other proxy connectivity tests can proceed.
Prepare Your Environment
Requirements for Hybrid Data Security
Cisco Webex License Requirements
To deploy Hybrid Data Security:
-
You must have Pro Pack for Cisco Webex Control Hub. (See https://www.cisco.com/go/pro-pack.)
Docker Desktop Requirements
Before you install your HDS nodes, you need Docker Desktop to run a setup program. Docker recently updated their licensing model. Your organization might require a paid subscription for Docker Desktop. For details, see the Docker blog post, " Docker is Updating and Extending Our Product Subscriptions".
X.509 Certificate Requirements
The certificate chain must meet the following requirements:
Requirement |
Details |
---|---|
|
By default, we trust the CAs in the Mozilla list (with the exception of WoSign and StartCom) at https://wiki.mozilla.org/CA:IncludedCAs. |
|
The CN does not need to be reachable or a live host. We recommend that you use a name which reflects your organization, for example, The CN must not contain a * (wildcard). The CN is used to verify the Hybrid Data Security nodes to Webex App clients. All of the Hybrid Data Security nodes in your cluster use the same certificate. Your KMS identifies itself using the CN domain, not any domain that is defined in the x.509v3 SAN fields. Once you have registered a node with this certificate, we do not support changing the CN domain name. Choose a domain that can apply to both the trial and production deployments. |
|
The KMS software does not support SHA1 signatures for validating connections to other organizations' KMSs. |
|
You can use a converter such as OpenSSL to change your certificate's format. You will need to enter the password when you run the HDS Setup Tool. |
The KMS software does not enforce key usage or extended key usage constraints. Some certificate authorities require that extended key usage constraints be applied to each certificate, such as server authentication. It is okay to use the server authentication or other settings.
Virtual Host Requirements
The virtual hosts that you will set up as Hybrid Data Security nodes in your cluster have the following requirements:
-
At least two separate hosts (3 recommended) colocated in the same secure data center
-
VMware ESXi 6.5 (or later) installed and running.
You must upgrade if you have an earlier version of ESXi.
-
Minimum 4 vCPUs, 8-GB main memory, 30-GB local hard disk space per server
Database server requirements
Create a new database for key storage. Don’t use the default database. The HDS applications, when installed, create the database schema.
There are two options for database server. The requirements for each are as follows:
PostgreSQL |
Microsoft SQL Server |
---|---|
|
|
Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage) |
Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage) |
The HDS software currently installs the following driver versions for communication with the database server:
PostgreSQL |
Microsoft SQL Server |
---|---|
Postgres JDBC driver 42.2.5 |
SQL Server JDBC driver 4.6 This driver version supports SQL Server Always On ( Always On Failover Cluster Instances and Always On availability groups). |
Additional requirements for Windows authentication against Microsoft SQL Server
If you want HDS nodes to use Windows authentication to gain access to your keystore database on Microsoft SQL Server, then you need the following configuration in your environment:
-
The HDS nodes, Active Directory infrastructure, and MS SQL Server must all be synchronized with NTP.
-
The Windows account you provide to HDS nodes must have read/write access to the database.
-
The DNS servers you provide to HDS nodes must be able to resolve your Key Distribution Center (KDC).
-
You may register the HDS database instance on your Microsoft SQL Server as a Service Principal Name (SPN) on your Active Directory. See Register a Service Principal Name for Kerberos Connections.
The HDS setup tool, HDS launcher, and local KMS all need to use Windows authentication to access the keystore database. They use the details from your ISO configuration to construct the SPN when requesting access with Kerberos authentication.
External connectivity requirements
Configure your firewall to allow the following connectivity for the HDS applications:
Application |
Protocol |
Port |
Direction from App |
Destination |
---|---|---|---|---|
Hybrid Data Security nodes |
TCP |
443 |
Outbound HTTPS and WSS |
|
HDS Setup Tool |
TCP |
443 |
Outbound HTTPS |
|
The Hybrid Data Security nodes work with network access translation (NAT) or behind a firewall, as long as the NAT or firewall allows the required outbound connections to the domain destinations in the preceding table. For connections going inbound to the Hybrid Data Security nodes, no ports should be visible from the internet. Within your data center, clients need access to the Hybrid Data Security nodes on TCP ports 443 and 22, for administrative purposes.
The URLs for the Common Identity (CI) hosts are region-specific. These are the current CI hosts:
Region |
Common Identity Host URLs |
---|---|
Americas |
|
European Union |
|
Canada |
|
Proxy Server Requirements
-
We officially support the following proxy solutions that can integrate with your Hybrid Data Security nodes.
-
Transparent proxy—Cisco Web Security Appliance (WSA).
-
Explicit proxy—Squid.
Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:) connections. To work around this issue, see Configure Squid Proxies for Hybrid Data Security.
-
-
We support the following authentication type combinations for explicit proxies:
-
No authentication with HTTP or HTTPS
-
Basic authentication with HTTP or HTTPS
-
Digest authentication with HTTPS only
-
-
For a transparent inspecting proxy or an HTTPS explicit proxy, you must have a copy of the proxy's root certificate. The deployment instructions in this guide tell you how to upload the copy to the Hybrid Data Security nodes' trust stores.
-
The network hosting the HDS nodes must be configured to force outbound TCP traffic on port 443 to route through the proxy.
-
Proxies that inspect web traffic may interfere with web socket connections. If this problem occurs, bypassing (not inspecting) traffic to
wbx2.com
andciscospark.com
will solve the problem.
Complete the Prerequisites for Hybrid Data Security
1 |
Make sure your Webex organization is enabled for Pro Pack for Cisco Webex Control Hub, and get the credentials of an account with full organization administrator rights. Contact your Cisco partner or account manager for help with this process. |
2 |
Choose a domain name for your HDS deployment (for example, |
3 |
Prepare identical virtual hosts that you will set up as Hybrid Data Security nodes in your cluster. You need at least two separate hosts (3 recommended) colocated in the same secure data center, which meet the requirements in Virtual Host Requirements. |
4 |
Prepare the database server that will act as the key data store for the cluster, according to the Database server requirements. The database server must be colocated in the secure data center with the virtual hosts. |
5 |
For quick disaster recovery, set up a backup environment in a different data center. The backup environment mirrors the production environment of VMs and a backup database server. For example, if production has 3 VMs running HDS nodes, the backup environment should have 3 VMs. |
6 |
Set up a syslog host to collect logs from the nodes in the cluster. Gather its network address and syslog port (default is UDP 514). |
7 |
Create a secure backup policy for the Hybrid Data Security nodes, the database server, and the syslog host. At a minimum, to prevent unrecoverable data loss, you must back up the database and the configuration ISO file generated for the Hybrid Data Security nodes. Because the Hybrid Data Security nodes store the keys used in encryption and decryption of content, failure to maintain an operational deployment will result in the UNRECOVERABLE LOSS of that content. Webex App clients cache their keys, so an outage may not be immediately noticeable but will become evident over time. While temporary outages are impossible to prevent, they are recoverable. However, complete loss (no backups available) of either the database or configuration ISO file will result in unrecoverable customer data. The operators of the Hybrid Data Security nodes are expected to maintain frequent backups of the database and the configuration ISO file, and be prepared to rebuild the Hybrid Data Security data center if a catastrophic failure occurs. |
8 |
Ensure that your firewall configuration allows connectivity for your Hybrid Data Security nodes as outlined in External connectivity requirements. |
9 |
Install Docker ( https://www.docker.com) on any local machine running a supported OS (Microsoft Windows 10 Professional or Enterprise 64-bit, or Mac OSX Yosemite 10.10.3 or above) with a web browser that can access it at http://127.0.0.1:8080. You use the Docker instance to download and run the HDS Setup Tool, which builds the local configuration information for all the Hybrid Data Security nodes. Your organization might need a Docker Desktop license. See Docker Desktop Requirements for more information. To install and run the HDS Setup Tool, the local machine must have the connectivity outlined in External connectivity requirements. |
10 |
If you're integrating a proxy with Hybrid Data Security, make sure that it meets the Proxy Server Requirements. |
11 |
If your organization uses directory synchronization, create a group in Active Directory called Keys for a given space are set by the creator of the space. When selecting pilot users, bear in mind that if you decide to permanently deactivate the Hybrid Data Security deployment, all users lose access to content in the spaces that were created by the pilot users. The loss becomes apparent as soon as users' apps refresh their cached copies of the content. |
Set up a Hybrid Data Security Cluster
Hybrid Data Security Deployment Task Flow
Before you begin
1 |
Perform initial set up and download installation files Download the OVA file to your local machine for later use. |
2 |
Create a Configuration ISO for the HDS Hosts Use the HDS Setup Tool to create an ISO configuration file for the Hybrid Data Security nodes. |
3 |
Create a virtual machine from the OVA file and perform initial configuration, such as network settings. The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions. |
4 |
Set up the Hybrid Data Security VM Sign in to the VM console and set the sign-in credentials. Configure the network settings for the node if you didn't configure them at the time of OVA deployment. |
5 |
Upload and Mount the HDS Configuration ISO Configure the VM from the ISO configuration file that you created with the HDS Setup Tool. |
6 |
Configure the HDS Node for Proxy Integration If the network environment requires proxy configuration, specify the type of proxy that you will use for the node, and add the proxy certificate to the trust store if needed. |
7 |
Register the First Node in the Cluster Register the VM with the Cisco Webex cloud as a Hybrid Data Security node. |
8 |
Create and Register More Nodes Complete the cluster setup. |
9 |
Run a Trial and Move to Production (next chapter) Until you start a trial, your nodes generate an alarm indicating that your service is not yet activated. |
Download Installation Files
1 |
Sign in to https://admin.webex.com, and then click Services. |
2 |
In the Hybrid Services section, find the Hybrid Data Security card, and then click Set up. If the card is disabled or you don’t see it, contact your account team or your partner organization. Give them your account number and ask to enable your organization for Hybrid Data Security. To find the account number, click the gear at the top right, next to your organization name. You can also download the OVA at any time from the Help section on the Settings page. On the Hybrid Data Security card, click Edit settings to open the page. Then, click Download Hybrid Data Security software in the Help section. Older versions of the software package (OVA) will not be compatible with the latest Hybrid Data Security upgrades. This can result in issues while upgrading the application. Make sure you download the latest version of the OVA file. |
3 |
Select No to indicate that you haven’t set up the node yet, and then click Next. The OVA file automatically begins to
download. Save the file to a location on your
machine.
|
4 |
Optionally, click Open Deployment Guide to check if there’s a later version of this guide available. |
Create a Configuration ISO for the HDS Hosts
The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.
Before you begin
-
The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Control Hub account with full administrator rights for your organization.
If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5. This table gives some possible environment variables:
Description
Variable
HTTP Proxy without authentication
GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT
HTTPS Proxy without authentication
GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT
HTTP Proxy with authentication
GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT
HTTPS Proxy with authentication
GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT
-
The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:
-
Database credentials
-
Certificate updates
-
Changes to authorization policy
-
-
If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.
1 |
At your machine's command line, enter the appropriate command for your environment: In regular environments:
In FedRAMP environments:
This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore. | ||||||||||
2 |
To sign in to the Docker image registry, enter the following:
| ||||||||||
3 |
At the password prompt, enter this hash:
| ||||||||||
4 |
Download the latest stable image for your environment: In regular environments:
In FedRAMP environments:
| ||||||||||
5 |
When the pull completes, enter the appropriate command for your environment:
When the container is running, you see "Express server listening on port 8080." | ||||||||||
6 |
The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost. Use a web browser to go to the localhost, The tool uses this first entry of the username to set the proper environment for that account. The tool then displays the standard sign-in prompt. | ||||||||||
7 |
When prompted, enter your Control Hub customer admin sign-in credentials, and then click Log in to allow access to the required services for Hybrid Data Security. | ||||||||||
8 |
On the Setup Tool overview page, click Get Started. | ||||||||||
9 |
On the ISO Import page, you have these options:
| ||||||||||
10 |
Check that your X.509 certificate meets the requirements in X.509 Certificate Requirements.
| ||||||||||
11 |
Enter the database address and account for HDS to access your key datastore: | ||||||||||
12 |
Select a TLS Database Connection Mode:
When you upload the root certificate (if necessary) and click Continue, the HDS Setup Tool tests the TLS connection to the database server. The tool also verifies the certificate signer and hostname, if applicable. If a test fails, the tool shows an error message describing the problem. You can choose whether to ignore the error and continue with the setup. (Because of connectivity differences, the HDS nodes might be able to establish the TLS connection even if the HDS Setup Tool machine can't successfully test it.) | ||||||||||
13 |
On the System Logs page, configure your Syslogd server: | ||||||||||
14 |
(Optional) You can change the default value for some database connection parameters in Advanced Settings. Generally, this parameter is the only one that you might want to change:
| ||||||||||
15 |
Click Continue on the Reset Service Accounts Password screen. Service account passwords have a nine-month lifespan. Use this screen when your passwords are nearing expiry or you want to reset them to invalidate previous ISO files. | ||||||||||
16 |
Click Download ISO File. Save the file in a location that's easy to find. | ||||||||||
17 |
Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes. | ||||||||||
18 |
To shut down the Setup tool, type |
What to do next
Back up the configuration ISO file. You need it to create more nodes for recovery, or to make configuration changes. If you lose all copies of the ISO file, you've also lost the master key. Recovering the keys from your PostgreSQL or Microsoft SQL Server database isn't possible.
We never have a copy of this key and can't help if you lose it.
Install the HDS Host OVA
1 |
Use the VMware vSphere client on your computer to log into the ESXi virtual host. |
2 |
Select File > Deploy OVF Template. |
3 |
In the wizard, specify the location of the OVA file that you downloaded earlier, and then click Next. |
4 |
On the Select a name and folder page, enter a Virtual machine name for the node (for example, "HDS_Node_1"), choose a location where the virtual machine node deployment can reside, and then click Next. |
5 |
On the Select a compute resource page, choose the destination compute resource, and then click Next. A validation check runs. After it finishes, the template details appear. |
6 |
Verify the template details and then click Next. |
7 |
If you are asked to choose the resource configuration on the Configuration page, click 4 CPU and then click Next. |
8 |
On the Select storage page, click Next to accept the default disk format and VM storage policy. |
9 |
On the Select networks page, choose the network option from the list of entries to provide the desired connectivity to the VM. |
10 |
On the Customize template page, configure the following network settings:
If preferred, you can skip the network setting configuration and follow the steps in Set up the Hybrid Data Security VM to configure the settings from the node console. The option to configure network settings during OVA deployment has been tested with ESXi 6.5. The option may not be available in earlier versions. |
11 |
Right-click the node VM, and then choose .The Hybrid Data Security software is installed as a guest on the VM Host. You are now ready to sign in to the console and configure the node. Troubleshooting Tips You may experience a delay of a few minutes before the node containers come up. A bridge firewall message appears on the console during first boot, during which you can't sign in. |
Set up the Hybrid Data Security VM
Use this procedure to sign in to the Hybrid Data Security node VM console for the first time and set the sign-in credentials. You can also use the console to configure the network settings for the node if you didn't configure them at the time of OVA deployment.
1 |
In the VMware vSphere client, select your Hybrid Data Security node VM and select the Console tab. The VM boots up and a login prompt appears. If the login prompt does not display, press Enter.
|
2 |
Use the following default login and password to sign in and change the credentials: Since you are signing in to your VM for the first time, you are required to change the administrator password. |
3 |
If you already configured the network settings in Install the HDS Host OVA, skip the rest of this procedure. Otherwise, in the main menu, select the Edit Configuration option. |
4 |
Set up a static configuration with IP address, Mask, Gateway and DNS information. Your node should have an internal IP address and DNS name. DHCP is not supported. |
5 |
(Optional) Change the hostname, domain or NTP server(s), if needed to match your network policy. You do not need to set the domain to match the domain that you used to obtain the X.509 certificate. |
6 |
Save the network configuration and reboot the VM so that the changes take effect. |
Upload and Mount the HDS Configuration ISO
Before you begin
Because the ISO file holds the master key, it should only be exposed on a "need to know" basis, for access by the Hybrid Data Security VMs and any administrators who might need to make changes. Make sure that only those administrators can access the datastore.
1 |
Upload the ISO file from your computer: |
2 |
Mount the ISO file: |
What to do next
If your IT policy requires, you can optionally unmount the ISO file after all your nodes pick up the configuration changes. See (Optional) Unmount ISO After HDS Configuration for details.
Configure the HDS Node for Proxy Integration
If the network environment requires a proxy, use this procedure to specify the type of proxy that you want to integrate with Hybrid Data Security. If you choose a transparent inspecting proxy or an HTTPS explicit proxy, you can use the node's interface to upload and install the root certificate. You can also check the proxy connection from the interface, and troubleshoot any potential issues.
Before you begin
-
See Proxy Support for an overview of the supported proxy options.
1 |
Enter the HDS node setup URL |
2 |
Go to Trust Store & Proxy, and then choose an option:
Follow the next steps for a transparent inspecting proxy, an HTTP explicit proxy with Basic authentication, or an HTTPS explicit proxy. |
3 |
Click Upload a Root Certificate or End Entity Certificate, and then navigate to a choose the root certificate for the proxy. The certificate is uploaded but not yet installed because you must reboot the node to install the certificate. Click the chevron arrow by the certificate issuer name to get more details or click Delete if you made a mistake and want to reupload the file. |
4 |
Click Check Proxy Connection to test the network connectivity between the node and the proxy. If the connection test fails, you'll see an error message that shows the reason and how you can correct the issue. If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server. This condition is expected in many explicit proxy configurations. You can continue with the setup, and the node will function in Blocked External DNS Resolution mode. If you think this is an error, complete these steps, and then see Turn off Blocked External DNS Resolution Mode. |
5 |
After the connection test passes, for explicit proxy set to https only, turn the toggle on to Route all port 443/444 https requests from this node through the explicit proxy. This setting requires 15 seconds to take effect. |
6 |
Click Install All Certificates Into the Trust Store (appears for an HTTPS explicit proxy or a transparent inspecting proxy) or Reboot (appears for an HTTP explicit proxy), read the prompt, and then click Install if you're ready. The node reboots within a few minutes. |
7 |
After the node reboots, sign in again if needed, and then open the Overview page to check the connectivity checks to make sure they are all in green status. The proxy connection check only tests a subdomain of webex.com. If there are connectivity problems, a common issue is that some of the cloud domains listed in the install instructions are being blocked at the proxy. |
Register the First Node in the Cluster
When you register your first node, you create a cluster to which the node is assigned. A cluster contains one or more nodes deployed to provide redundancy.
Before you begin
-
Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.
-
Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.
1 |
Sign in to https://admin.webex.com. |
2 |
From the menu on the left side of the screen, select Services. |
3 |
In the Hybrid Services section, find Hybrid Data Security and click Set up. The Register Hybrid Data Security Node page appears.
|
4 |
Select Yes to indicate that you have set up the node and are ready to register it, and then click Next. |
5 |
In the first field, enter a name for the cluster to which you want to assign your Hybrid Data Security node. We recommend that you name a cluster based on where the nodes of the cluster are located geographically. Examples: "San Francisco" or "New York" or "Dallas" |
6 |
In the second field, enter the internal IP address or fully qualified domain name (FQDN) of your node and click Next. This IP address or FQDN should match the IP address or hostname and domain that you used in Set up the Hybrid Data Security VM. A message appears indicating you can register your node to the Webex.
|
7 |
Click Go to Node. |
8 |
Click Continue in the warning message. After a few moments, you are redirected to the node connectivity tests for Webex services. If all tests are successful, the Allow Access to Hybrid Data Security Node page appears. There, you confirm that you want to give permissions to your Webex organization to access your node.
|
9 |
Check the Allow Access to Your Hybrid Data Security Node checkbox, and then click Continue. Your account is validated and the "Registration Complete" message indicates that your node is now registered to the Webex cloud.
|
10 |
Click the link or close the tab to go back to the Control Hub Hybrid Data Security page. On the Hybrid Data Security page, the new cluster containing the node that you registered is displayed. The node will automatically download the latest software from the cloud.
|
Create and Register More Nodes
At this time, the backup VMs that you created in Complete the Prerequisites for Hybrid Data Security are standby hosts which are only used in the event of disaster recovery; they are not registered with the system until then. For details, see Disaster Recovery using Standby Data Center.
Before you begin
-
Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.
-
Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.
1 |
Create a new virtual machine from the OVA, repeating the steps in Install the HDS Host OVA. |
2 |
Set up the initial configuration on the new VM, repeating the steps in Set up the Hybrid Data Security VM. |
3 |
On the new VM, repeat the steps in Upload and Mount the HDS Configuration ISO. |
4 |
If you are setting up a proxy for your deployment, repeat the steps in Configure the HDS Node for Proxy Integration as needed for the new node. |
5 |
Register the node. Your node is registered. Note that until you start a trial, your nodes generate an alarm indicating that your service is not yet activated.
|
What to do next
Run a Trial and Move to Production
Trial to Production Task Flow
After you set up a Hybrid Data Security cluster, you can start a pilot, add users to it, and begin using it for testing and verifying your deployment in preparation for moving to production.
Before you begin
1 |
If applicable, synchronize the If your organization uses directory synchronization for users, you must select the |
2 |
Start a trial. Until you do this task, your nodes generate an alarm indicating that the service is not yet activated. |
3 |
Test Your Hybrid Data Security Deployment Check that key requests are passing to your Hybrid Data Security deployment. |
4 |
Monitor Hybrid Data Security Health Check status, and set up email notifications for alarms. |
5 | |
6 |
Complete the trial phase with one of the following actions: |
Activate Trial
Before you begin
If your organization uses directory synchronization for users, you must select the HdsTrialGroup
group object for synchronization to the cloud before you can start a trial for your organization. For instructions, see the Deployment Guide for Cisco Directory Connector.
1 |
Sign in to https://admin.webex.com, and then select Services. |
2 |
Under Hybrid Data Security, click Settings. |
3 |
In the Service Status section, click Start Trial. The service status changes to trial mode.
|
4 |
Click Add Users and enter the email address of one or more users to pilot using your Hybrid Data Security nodes for encryption and indexing services. (If your organization uses directory synchronization, use Active Directory to manage the trial group, |
Test Your Hybrid Data Security Deployment
Before you begin
-
Set up your Hybrid Data Security deployment.
-
Activate the trial, and add several trial users.
-
Ensure that you have access to the syslog to verify that key requests are passing to your Hybrid Data Security deployment.
1 |
Keys for a given space are set by the creator of the space. Sign in to the Webex App as one of the pilot users, and then create a space and invite at least one pilot user and one non-pilot user. If you deactivate the Hybrid Data Security deployment, content in spaces that pilot users create is no longer accessible once the client-cached copies of the encryption keys are replaced. |
2 |
Send messages to the new space. |
3 |
Check the syslog output to verify that the key requests are passing to your Hybrid Data Security deployment. |
Monitor Hybrid Data Security Health
1 |
In Control Hub, select Services from the menu on the left side of the screen. |
2 |
In the Hybrid Services section, find Hybrid Data Security and click Settings. The Hybrid Data Security Settings page appears.
|
3 |
In the Email Notifications section, type one or more email addresses separated by commas, and press Enter. |
Add or Remove Users from Your Trial
If you remove a user from the trial, the user's client will request keys and key creation from the cloud KMS instead of your KMS. If the client needs a key that is stored on your KMS, the cloud KMS will fetch it on the user's behalf.
If your organization uses directory synchronization, use Active Directory (instead of this procedure) to manage the trial group, HdsTrialGroup
; you can view the group members in Control Hub but cannot add or remove them.
1 |
Sign in to Control Hub, and then select Services. |
2 |
Under Hybrid Data Security, click Settings. |
3 |
In the Trial Mode section of the Service Status area, click Add Users, or click view and edit to remove users from the trial. |
4 |
Enter the email address of one or more users to add, or click the X by a user ID to remove the user from the trial. Then click Save. |
Move from Trial to Production
1 |
Sign in to Control Hub, and then select Services. |
2 |
Under Hybrid Data Security, click Settings. |
3 |
In the Service Status section, click Move to Production. |
4 |
Confirm that you want to move all of your users to production. |
End Your Trial Without Moving to Production
1 |
Sign in to Control Hub, and then select Services. |
2 |
Under Hybrid Data Security, click Settings. |
3 |
In the Deactivate section, click Deactivate. |
4 |
Confirm that you want to deactivate the service and end the trial. |
Manage your HDS Deployment
Manage HDS Deployment
Use the tasks described here to manage your Hybrid Data Security deployment.
Set Cluster Upgrade Schedule
To set the upgrade schedule:
1 |
Sign in to Control Hub. |
2 |
On the Overview page, under Hybrid Services, select Hybrid Data Security. |
3 |
On the Hybrid Data Security Resources page, select the cluster. |
4 |
In the Overview panel on the right, under Cluster Settings, select the cluster name. |
5 |
On the Settings page, under Upgrade, select the time and time zone for the upgrade schedule. Note: Under the time zone, the next available upgrade date and time is displayed. You can postpone the upgrade to the following day, if needed, by clicking Postpone. |
Change the Node Configuration
-
Changing x.509 certificates due to expiration or other reasons.
We don't support changing the CN domain name of a certificate. The domain must match the original domain used to register the cluster.
-
Updating database settings to change to a replica of the PostgreSQL or Microsoft SQL Server database.
We don’t support migrating data from PostgreSQL to Microsoft SQL Server, or the opposite way. To switch the database environment, start a new deployment of Hybrid Data Security.
-
Creating a new configuration to prepare a new data center.
Also, for security purposes, Hybrid Data Security uses service account passwords that have a nine-month lifespan. After the HDS Setup tool generates these passwords, you deploy them to each of your HDS nodes in the ISO config file. When your organization's passwords are nearing expiration, you receive a notice from the Webex team to reset the password for your machine account. (The email includes the text, "Use the machine account API to update the password.") If your passwords haven't expired yet, the tool gives you two options:
-
Soft reset—The old and new passwords both work for up to 10 days. Use this period to replace the ISO file on the nodes gradually.
-
Hard reset—The old passwords stop working immediately.
If your passwords expire without a reset, it impacts your HDS service, requiring an immediate hard reset and replacement of the ISO file on all nodes.
Use this procedure to generate a new configuration ISO file and apply it to your cluster.
Before you begin
-
The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Control Hub account with full administrator rights for your organization.
If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in 1.e. This table gives some possible environment variables:
Description
Variable
HTTP Proxy without authentication
GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT
HTTPS Proxy without authentication
GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT
HTTP Proxy with authentication
GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT
HTTPS Proxy with authentication
GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT
-
You need a copy of the current configuration ISO file to generate a new configuration. The ISO contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the ISO when you make configuration changes, including database credentials, certificate updates, or changes to authorization policy.
1 |
Using Docker on a local machine, run the HDS Setup Tool. |
2 |
If you only have one HDS node running, create a new Hybrid Data Security node VM and register it using the new configuration ISO file. For more detailed instructions, see Create and Register More Nodes. |
3 |
For existing HDS nodes that are running the older configuration file, mount the ISO file. Perform the following procedure on each node in turn, updating each node before turning off the next node: |
4 |
Repeat step 3 to replace the configuration on each remaining node that is running the old configuration. |
Turn off Blocked External DNS Resolution Mode
When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. If the node's DNS server can't resolve public DNS names, the node automatically goes into Blocked External DNS Resolution mode.
If your nodes are able to resolve public DNS names through internal DNS servers, you can turn off this mode by rerunning the proxy connection test on each node.
Before you begin
1 |
In a web browser, open the Hybrid Data Security node interface (IP address/setup, for example, https://192.0.2.0/setup), enter the admin credentials you set up for the node, and then click Sign In. |
2 |
Go to Overview (the default page). When enabled, Blocked External DNS Resolution is set to Yes. |
3 |
Go to the Trust Store & Proxy page. |
4 |
Click Check Proxy Connection. If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server and will remain in this mode. Otherwise, after you reboot the node and go back to the Overview page, Blocked External DNS Resolution should be set to no. |
What to do next
Remove a Node
1 |
Use the VMware vSphere client on your computer to log into the ESXi virtual host and power off the virtual machine. |
2 |
Remove the node: |
3 |
In the vSphere client, delete the VM. (In the left navigation pane, right-click on the VM and click Delete.) If you don’t delete the VM, remember to unmount the configuration ISO file. Without the ISO file, you can't use the VM to access your security data. |
Disaster Recovery using Standby Data Center
The most critical service that your Hybrid Data Security cluster provides is the creation and storage of keys used to encrypt messages and other content stored in the Webex cloud. For each user within the organization who is assigned to Hybrid Data Security, new key creation requests are routed to the cluster. The cluster is also responsible for returning the keys that it's created to any users authorized to retrieve them, for example, members of a conversation space.
Because the cluster performs the critical function of providing these keys, it's imperative that the cluster remains running and that proper backups are maintained. Loss of the Hybrid Data Security database or of the configuration ISO used for the schema will result in UNRECOVERABLE LOSS of customer content. The following practices are mandatory to prevent such a loss:
If a disaster causes the HDS deployment in the primary data center to become unavailable, follow this procedure to manually failover to the standby data center.
1 |
Start the HDS Setup tool and follow the steps mentioned in Create a Configuration ISO for the HDS Hosts. |
2 |
After configuring the Syslogd server, click on Advanced Settings |
3 |
On the Advanced Settings page, add the configuration below or remove the
|
4 |
Complete the configuration process and save the ISO file in a location that's easy to find. |
5 |
Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes. |
6 |
In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.. |
7 |
Click Edit Settings >CD/DVD Drive 1 and select Datastore ISO File. Make sure Connected and Connect at power on are checked so that updated configuration changes can take effect after starting the nodes. |
8 |
Power on the HDS node and make sure there are no alarms for at least 15 minutes. |
9 |
Repeat the process for every node in the standby data center. Check the syslog output to verify that the nodes of the standby data center are not in passive mode. “KMS configured in passive mode” should not appear in the syslogs. |
What to do next
(Optional) Unmount ISO After HDS Configuration
The standard HDS configuration runs with the ISO mounted. But, some customers prefer not leaving ISO files continuously mounted. You can unmount the ISO file after all HDS nodes pick up the new configuration.
You still use the ISO files to make configuration changes. When you create a new ISO or update an ISO through the Setup Tool, you must mount the updated ISO on all your HDS nodes. Once all your nodes have picked up the configuration changes, you can unmount the ISO again with this procedure.
Before you begin
Upgrade all your HDS nodes to version 2021.01.22.4720 or later.
1 |
Shut down one of your HDS nodes. |
2 |
In the vCenter Server Appliance, select the HDS node. |
3 |
Choose Datastore ISO File. and uncheck |
4 |
Power on the HDS node and ensure there are no alarms for atleast 20 minutes. |
5 |
Repeat for each HDS node in turn. |
Troubleshoot Hybrid Data Security
View Alerts and Troubleshoot
A Hybrid Data Security deployment is considered unavailable if all nodes in the cluster are unreachable, or the cluster is working so slowly that requests time out. If users cannot reach your Hybrid Data Security cluster, they experience the following symptoms:
-
New spaces cannot be created (unable to create new keys)
-
Messages and space titles fail to decrypt for:
-
New users added to a space (unable to fetch keys)
-
Existing users in a space using a new client (unable to fetch keys)
-
-
Existing users in a space will continue to run successfully as long as their clients have a cache of the encryption keys
It's important that you properly monitor your Hybrid Data Security cluster and address any alerts promptly to avoid disruption of service.
Alerts
If there is a problem with the Hybrid Data Security setup, Control Hub displays alerts to the organization administrator, and sends emails to the configured email address. The alerts cover many common scenarios.
Alert |
Action |
---|---|
Local database access failure. |
Check for database errors or local network issues. |
Local database connection failure. |
Check that the database server is available, and the right service account credentials were used in node configuration. |
Cloud service access failure. |
Check that the nodes can access the Webex servers as specified in External connectivity requirements. |
Renewing cloud service registration. |
Registration to cloud services was dropped. Renewal of registration is in progress. |
Cloud service registration dropped. |
Registration to cloud services terminated. Service is shutting down. |
Service not yet activated. |
Activate a trial, or finish moving the trial to production. |
Configured domain does not match server certificate. |
Ensure that your server certificate matches the configured service activation domain. The most likely cause is that the certificate CN was recently changed and is now different from the CN that was used during initial setup. |
Failed to authenticate to cloud services. |
Check for accuracy and possible expiration of service account credentials. |
Failed to open local keystore file. |
Check for integrity and password accuracy on local keystore file. |
Local server certificate is invalid. |
Check the server certificate's expiration date and confirm that it was issued by a trusted Certificate Authority. |
Unable to post metrics. |
Check local network access to external cloud services. |
/media/configdrive/hds directory does not exist. |
Check the ISO mount configuration on virtual host. Verify that the ISO file exists, that it is configured to mount on reboot, and that it mounts successfully. |
Troubleshoot Hybrid Data Security
1 |
Review Control Hub for any alerts and fix any items you find there. |
2 |
Review the syslog server output for activity from the Hybrid Data Security deployment. |
3 |
Contact Cisco support. |
Other Notes
Known Issues for Hybrid Data Security
-
If you shut down your Hybrid Data Security cluster (by deleting it in Control Hub or by shutting down all nodes), lose your configuration ISO file, or lose access to the keystore database, your Webex App users can no longer use spaces under their People list that were created with keys from your KMS. This applies to both trial and production deployments. We do not currently have a workaround or fix for this issue and urge you not to shut down your HDS services once they are handling active user accounts.
-
A client which has an existing ECDH connection to a KMS maintains that connection for a period of time (likely one hour). When a user becomes a member of a Hybrid Data Security trial, the user's client continues to use the existing ECDH connection until it times out. Alternatively, the user can sign out and back in to the Webex App app to update the location that the app contacts for encryption keys.
The same behavior occurs when you move a trial to production for the organization. All non-trial users with existing ECDH connections to the previous data security services will continue to use those services until the ECDH connection is renegotiated (through timeout or by signing out and back in).
Use OpenSSL to Generate a PKCS12 File
Before you begin
-
OpenSSL is one tool that can be used to make the PKCS12 file in the proper format for loading in the HDS Setup Tool. There are other ways to do this, and we do not support or promote one way over another.
-
If you do choose to use OpenSSL, we are providing this procedure as a guideline to help you create a file that meets the X.509 certificate requirements in X.509 Certificate Requirements. Understand those requirements before you continue.
-
Install OpenSSL in a supported environment. See https://www.openssl.org for the software and documentation.
-
Create a private key.
-
Start this procedure when you receive the server certificate from your Certificate Authority (CA).
1 |
When you receive the server certificate from your CA, save it as |
2 |
Display the certificate as text, and verify the details.
|
3 |
Use a text editor to create a certificate bundle file called
|
4 |
Create the .p12 file with the friendly name
|
5 |
Check the server certificate details. |
What to do next
Return to Complete the Prerequisites for Hybrid Data Security. You will use the hdsnode.p12
file,
and the password you've set for it, in Create a Configuration ISO for the HDS Hosts.
You can reuse these files to request a new certificate when the original certificate expires.
Traffic between the HDS Nodes and the Cloud
Outbound Metrics Collection Traffic
The Hybrid Data Security nodes send certain metrics to the Webex cloud. These include system metrics for heap max, heap used, CPU load, and thread count; metrics on synchronous and asynchronous threads; metrics on alerts involving a threshold of encryption connections, latency, or a request queue length; metrics on the datastore; and encryption connection metrics. The nodes send encrypted key material over an out-of-band (separate from the request) channel.
Inbound Traffic
The Hybrid Data Security nodes receive the following types of inbound traffic from the Webex cloud:
-
Encryption requests from clients, which are routed by the encryption service
-
Upgrades to the node software
Configure Squid Proxies for Hybrid Data Security
Websocket Cannot Connect Through Squid Proxy
Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:
) connections that Hybrid Data Security requires. These sections give guidance on how to configure various versions of Squid to ignore wss:
traffic for proper operation of the services.
Squid 4 and 5
Add the on_unsupported_protocol
directive to squid.conf
:
on_unsupported_protocol tunnel all
Squid 3.5.27
We successfully tested Hybrid Data Security with the following rules added to squid.conf
. These rules are subject to change as we develop features and update the Webex cloud.
acl wssMercuryConnection ssl::server_name_regex mercury-connection
ssl_bump splice wssMercuryConnection
acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump stare step2 all
ssl_bump bump step3 all