Deployment Guide for Multi-Tenant Hybrid Data Security (HDS) (Beta)

In this article

Feedback?

New and changed information

This table covers new features or functionality, changes to existing content, and any major errors that were fixed in the Deployment Guide for Multi-Tenant Hybrid Data Security.

Date	Changes made
March 04, 2025	Added Run HDS Setup tool using Podman Desktop section. Added a note in Docker Desktop Requirements providing customers with an alternative open-source option. Updated Create a Configuration ISO for the HDS Hosts and Change the Node Configuration with instructions to use Podman Desktop for customers without a Docker Desktop license.
January 30, 2025	Added SQL server version 2022 to the list of supported SQL servers in Database server requirements.
January 15, 2025	Added Limitations of Multi-Tenant Hybrid Data Security.
January 08, 2025	Added a note in Perform initial set up and download installation files stating that clicking Set up on the HDS card in Partner Hub is an important step of the installation process.
January 07, 2025	Updated Virtual Host Requirements, Hybrid Data Security Deployment Task Flow, and Install the HDS Host OVA to show new requirement of ESXi 7.0.
December 13, 2024	First published.

Get started with Multi-Tenant Hybrid Data Security

Multi-Tenant Hybrid Data Security Overview

From day one, data security has been the primary focus in designing Webex App. The cornerstone of this security is end-to-end content encryption, enabled by Webex App clients interacting with the Key Management Service (KMS). The KMS is responsible for creating and managing the cryptographic keys that clients use to dynamically encrypt and decrypt messages and files.

By default, all Webex App customers get end-to-end encryption with dynamic keys stored in the cloud KMS, in Cisco's security realm. Hybrid Data Security moves the KMS and other security-related functions to your enterprise data center, so nobody but you holds the keys to your encrypted content.

Multi-Tenant Hybrid Data Security enables organizations to leverage HDS through a trusted local partner, who can act as a service provider and manage on-premises encryption and other security services. This setup allows the partner organization to have complete control over deployment and management of encryption keys and ensures user data of customer organizations is safe from external access. Partner organizations set up HDS instances and create HDS clusters as needed. Each instance can support multiple customer organizations unlike a regular HDS deployment which is limited to a single organization.

While partner organizations have control over deployment and management, they do not have access to data and content generated by customers. This access is limited to customer organizations and their users.

This also allows smaller organizations to leverage HDS, since Key management service and security infrastructure like data centers are owned by the trusted local partner.

How Multi-Tenant Hybrid Data Security provides data sovereignty and data control

User generated content is protected from external access, like cloud service providers.
Local trusted partners manage the encryption keys of customers with whom they already have an established relationship.
Option for local technical support, if provided by the partner.
Supports Meetings, Messaging and Calling content.

This document is aimed at aiding partner organizations to set up and manage customers under a Multi-Tenant Hybrid Data Security system.

Limitations of Multi-Tenant Hybrid Data Security

Partner organizations must not have any existing HDS deployment active in Control Hub.
Tenant or customer organizations that wish to be managed by a partner must not have any existing HDS deployment in Control Hub.
Once Multi-Tenant HDS is deployed by the partner, all users of customer organizations as well as users of the partner organization begin leveraging Multi-Tenant HDS for their encryption services.

The partner organization and customer organizations they manage will be on the same Multi-Tenant HDS deployment.

The partner organization will no longer use cloud KMS after Multi-Tenant HDS is deployed.
There is no mechanism to move keys back to Cloud KMS after an HDS deployment.
Currently, each Multi-Tenant HDS deployment can have only one cluster, with multiple nodes under it.
Administrator roles have certain limitations; see the section below for details.

Roles in Multi-Tenant Hybrid Data Security

Partner full Administrator - Can manage settings for all customers that the partner manages. Can also assign admin roles to existing users in the organization and assign specific customers to be managed by partner admins.
Partner administrator - Can manage settings for customers that the administrator provisioned or who have been assigned to the user.
Full administrator - Administrator of the partner organization who is authorized to carry out tasks such as modifying organization settings, managing licenses, and assigning roles.

End to end Multi-Tenant HDS setup and management of all customer organizations - Partner full administrator and Full administrator rights required.
Management of assigned tenant organizations - Partner administrator and Full administrator rights required.

Security Realm Architecture

The Webex cloud architecture separates different types of service into separate realms, or trust domains, as depicted below.

***Realms of Separation (without Hybrid Data Security)***

To further understand Hybrid Data Security, let's first look at this pure cloud case, where Cisco is providing all functions in its cloud realms. The identity service, the only place where users can be directly correlated with their personal information such as email address, is logically and physically separate from the security realm in data center B. Both are in turn separate from the realm where encrypted content is ultimately stored, in data center C.

In this diagram, the client is the Webex App running on a user's laptop, and has authenticated with the identity service. When the user composes a message to send to a space, the following steps take place:

The client establishes a secure connection with the key management service (KMS), then requests a key to encrypt the message. The secure connection uses ECDH, and the KMS encrypts the key using an AES-256 master key.
The message is encrypted before it leaves the client. The client sends it to the indexing service, which creates encrypted search indexes to aid in future searches for the content.
The encrypted message is sent to the compliance service for compliance checks.
The encrypted message is stored in the storage realm.

When you deploy Hybrid Data Security, you move the security realm functions (KMS, indexing, and compliance) to your on-premises data center. The other cloud services that make up Webex (including identity and content storage) remain in Cisco’s realms.

Collaborating with Other Organizations

Users in your organization may regularly use Webex App to collaborate with external participants in other organizations. When one of your users requests a key for a space that is owned by your organization (because it was created by one of your users) your KMS sends the key to the client over an ECDH secured channel. However, when another organization owns the key for the space, your KMS routes the request out to the Webex cloud through a separate ECDH channel to get the key from the appropriate KMS, and then returns the key to your user on the original channel.

The KMS service running on Org A validates the connections to KMSs in other organizations using x.509 PKI certificates. See Prepare your Environment for details on generating an x.509 certificate to use with your Multi-Tenant Hybrid Data Security deployment.

Expectations for Deploying Hybrid Data Security

A Hybrid Data Security deployment requires significant commitment and an awareness of the risks that come with owning encryption keys.

To deploy Hybrid Data Security, you must provide:

A secure data center in a country that is a supported location for the Cisco Webex Teams plans.
The equipment, software, and network access described in Prepare your Environment.

Complete loss of either the configuration ISO that you build for Hybrid Data Security or the database that you provide will result in the loss of the keys. Key loss prevents users from decrypting space content and other encrypted data in Webex App. If this happens, you can build a new deployment, but only new content will be visible. To avoid loss of access to data, you must:

Manage the backup and recovery of the database and the configuration ISO.
Be prepared to perform quick disaster recovery if a catastrophe occurs, such as database disk failure or data center disaster.

There is no mechanism to move keys back to the Cloud after an HDS deployment.

High-level Setup process

This document covers the setup and management of a Multi-Tenant Hybrid Data Security deployment:

Set up Hybrid Data Security—This includes preparing required infrastructure and installing Hybrid Data Security software, building an HDS cluster, adding tenant organizations to the cluster and managing their Customer Main Keys(CMKs). This will enable all users of your customer organizations to use your Hybrid Data Security cluster for security functions.

The setup, activation, and management phases are covered in detail in the next three chapters.
Maintain your Hybrid Data Security deployment—The Webex cloud automatically provides ongoing upgrades. Your IT department can provide tier one support for this deployment, and engage Cisco support as needed. You can use on-screen notifications and set up email-based alerts in Partner Hub.
Understand common alerts, troubleshooting steps, and known issues—If you run into trouble deploying or using Hybrid Data Security, the last chapter of this guide and the Known Issues appendix may help you determine and fix the issue.

Hybrid Data Security Deployment Model

Within your enterprise data center, you deploy Hybrid Data Security as a single cluster of nodes on separate virtual hosts. The nodes communicate with the Webex cloud through secure websockets and secure HTTP.

During the installation process, we provide you with the OVA file to set up the virtual appliance on the VMs that you provide. You use the HDS Setup Tool to create a custom cluster configuration ISO file that you mount on each node. The Hybrid Data Security cluster uses your provided Syslogd server and PostgreSQL or Microsoft SQL Server database. (You configure the Syslogd and database connection details in the HDS Setup Tool.)

Hybrid Data Security Deployment Model

The minimum number of nodes you can have in a cluster is two. We recommend at least three per cluster. Having multiple nodes ensures that service is not interrupted during a software upgrade or other maintenance activity on a node. (The Webex cloud only upgrades one node at a time.)

All nodes in a cluster access the same key datastore, and log activity to the same syslog server. The nodes themselves are stateless, and handle key requests in round-robin fashion, as directed by the cloud.

Nodes become active when you register them in Partner Hub. To take an individual node out of service, you can deregister it, and later reregister it if needed.

Standby Data Center for Disaster Recovery

During deployment, you set up a secure standby data center. In the event of a data center disaster, you can manually fail your deployment over to the standby data center.

Before failover, Data Center A has active HDS nodes and the primary PostgreSQL or Microsoft SQL Server database, while B has a copy of the ISO file with additional configurations, VMs that are registered to the organization, and a standby database. After failover, Data Center B has active HDS nodes and the primary database, while A has unregistered VMs and a copy of the ISO file, and the database is in standby mode. — ***Manual Failover to Standby Data Center***

The databases of the active and standby data centers are in sync with each other which will minimize the time taken to perform the failover.

The active Hybrid Data Security nodes must always be in the same data center as the active database server.

Proxy Support

Hybrid Data Security supports explicit, transparent inspecting, and non-inspecting proxies. You can tie these proxies to your deployment so that you can secure and monitor traffic from the enterprise out to the cloud. You can use a platform admin interface on the nodes for certificate management and to check the overall connectivity status after you set up the proxy on the nodes.

The Hybrid Data Security nodes support the following proxy options:

No proxy—The default if you do not use the HDS node setup Trust Store & Proxy configuration to integrate a proxy. No certificate update is required.
Transparent non-inspecting proxy—The nodes are not configured to use a specific proxy server address and should not require any changes to work with a non-inspecting proxy. No certificate update is required.
Transparent tunneling or inspecting proxy—The nodes are not configured to use a specific proxy server address. No HTTP or HTTPS configuration changes are necessary on the nodes. However, the nodes need a root certificate so that they trust the proxy. Inspecting proxies are typically used by IT to enforce policies on which websites can be visited and which types of content are not permitted. This type of proxy decrypts all your traffic (even HTTPS).
Explicit proxy—With explicit proxy, you tell the HDS nodes which proxy server and authentication scheme to use. To configure an explicit proxy, you must enter the following information on each node:
1. Proxy IP/FQDN—Address that can be used to reach the proxy machine.
2. Proxy Port—A port number that the proxy uses to listen for proxied traffic.
3. Proxy Protocol—Depending on what your proxy server supports, choose between the following protocols:
  - HTTP—Views and controls all requests that the client sends.
  - HTTPS—Provides a channel to the server. The client receives and validates the server's certificate.
4. Authentication Type—Choose from among the following authentication types:
  - None—No further authentication is required.
    
    Available if you select either HTTP or HTTPS as the proxy protocol.
  - Basic—Used for an HTTP User Agent to provide a user name and password when making a request. Uses Base64 encoding.
    
    Available if you select either HTTP or HTTPS as the proxy protocol.
    
    Requires you to enter the user name and password on each node.
  - Digest—Used to confirm the account before sending sensitive information. Applies a hash function on the user name and password before sending over the network.
    
    Available only if you select HTTPS as the proxy protocol.
    
    Requires you to enter the user name and password on each node.

Example of Hybrid Data Security Nodes and Proxy

This diagram shows an example connection between the Hybrid Data Security, network and a proxy. For the transparent inspecting and HTTPS explicit inspecting proxy options, the same root certificate must be installed on the proxy and on the Hybrid Data Security nodes.

Blocked External DNS Resolution Mode (Explicit Proxy Configurations)

When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. In deployments with explicit proxy configurations that do not allow external DNS resolution for internal clients, if the node can't query the DNS servers, it automatically goes into Blocked External DNS Resolution mode. In this mode, node registration and other proxy connectivity tests can proceed.

Prepare your environment

Requirements for Multi-Tenant Hybrid Data Security

Cisco Webex License Requirements

To deploy Multi-Tenant Hybrid Data Security:

Partner Organizations: Contact your Cisco partner or account manager and ensure that the Multi-Tenant feature is enabled.
Tenant Organizations: You must have Pro Pack for Cisco Webex Control Hub. (See https://www.cisco.com/go/pro-pack.)

Docker Desktop Requirements

Before you install your HDS nodes, you need Docker Desktop to run a setup program. Docker recently updated their licensing model. Your organization might require a paid subscription for Docker Desktop. For details, see the Docker blog post, " Docker is Updating and Extending Our Product Subscriptions".

Customers without a Docker Desktop license can use an open source container management tool like Podman Desktop to run, manage, and create containers. See Run HDS Setup tool using Podman Desktop for details.

X.509 Certificate Requirements

The certificate chain must meet the following requirements:

Table 1. X.509 Certificate Requirements for Hybrid Data Security Deployment
Requirement	Details
Signed by a trusted Certificate Authority (CA)	By default, we trust the CAs in the Mozilla list (with the exception of WoSign and StartCom) at https://wiki.mozilla.org/CA:IncludedCAs.
Bears a Common Name (CN) domain name that identifies your Hybrid Data Security deployment Is not a wildcard certificate	The CN does not need to be reachable or a live host. We recommend that you use a name which reflects your organization, for example, `hds.company.com`. The CN must not contain a * (wildcard). The CN is used to verify the Hybrid Data Security nodes to Webex App clients. All of the Hybrid Data Security nodes in your cluster use the same certificate. Your KMS identifies itself using the CN domain, not any domain that is defined in the x.509v3 SAN fields. Once you have registered a node with this certificate, we do not support changing the CN domain name.
Non-SHA1 signature	The KMS software does not support SHA1 signatures for validating connections to other organizations' KMSs.
Formatted as a password-protected PKCS #12 file Use the friendly name of `kms-private-key` to tag the certificate, private key, and any intermediate certificates to upload.	You can use a converter such as OpenSSL to change your certificate's format. You will need to enter the password when you run the HDS Setup Tool.

The KMS software does not enforce key usage or extended key usage constraints. Some certificate authorities require that extended key usage constraints be applied to each certificate, such as server authentication. It is okay to use the server authentication or other settings.

Virtual Host Requirements

The virtual hosts that you will set up as Hybrid Data Security nodes in your cluster have the following requirements:

At least two separate hosts (3 recommended) colocated in the same secure data center
VMware ESXi 7.0 (or later) installed and running.

You must upgrade if you have an earlier version of ESXi.
Minimum 4 vCPUs, 8-GB main memory, 30-GB local hard disk space per server

Database server requirements

Create a new database for key storage. Don’t use the default database. The HDS applications, when installed, create the database schema.

There are two options for database server. The requirements for each are as follows:

Table 2. Database server requirements by type of database
PostgreSQL	Microsoft SQL Server
PostgreSQL 14, 15, or 16, installed and running.	SQL Server 2016, 2017, 2019, or 2022 (Enterprise or Standard) installed. SQL Server 2016 requires Service Pack 2 and Cumulative Update 2 or later.
Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage)	Minimum 8 vCPUs, 16-GB main memory, sufficient hard disk space and monitoring to ensure that it is not exceeded (2-TB recommended if you want to run the database for a long time without needing to increase the storage)

The HDS software currently installs the following driver versions for communication with the database server:

PostgreSQL	Microsoft SQL Server
Postgres JDBC driver 42.2.5	SQL Server JDBC driver 4.6 This driver version supports SQL Server Always On ( Always On Failover Cluster Instances and Always On availability groups).

PostgreSQL

Microsoft SQL Server

Postgres JDBC driver 42.2.5

SQL Server JDBC driver 4.6

This driver version supports SQL Server Always On ( Always On Failover Cluster Instances and Always On availability groups).

Additional requirements for Windows authentication against Microsoft SQL Server

If you want HDS nodes to use Windows authentication to gain access to your keystore database on Microsoft SQL Server, then you need the following configuration in your environment:

The HDS nodes, Active Directory infrastructure, and MS SQL Server must all be synchronized with NTP.
The Windows account you provide to HDS nodes must have read/write access to the database.
The DNS servers you provide to HDS nodes must be able to resolve your Key Distribution Center (KDC).
You may register the HDS database instance on your Microsoft SQL Server as a Service Principal Name (SPN) on your Active Directory. See Register a Service Principal Name for Kerberos Connections.

The HDS setup tool, HDS launcher, and local KMS all need to use Windows authentication to access the keystore database. They use the details from your ISO configuration to construct the SPN when requesting access with Kerberos authentication.

External connectivity requirements

Configure your firewall to allow the following connectivity for the HDS applications:

Application	Protocol	Port	Direction from App	Destination
Hybrid Data Security nodes	TCP	443	Outbound HTTPS and WSS	Webex servers: .wbx2.com .ciscospark.com All Common Identity hosts Other URLs that are listed for Hybrid Data Security in the Additional URLs for Webex Hybrid Services table of Network Requirements for Webex Services
HDS Setup Tool	TCP	443	Outbound HTTPS	*.wbx2.com All Common Identity hosts hub.docker.com

The Hybrid Data Security nodes work with network access translation (NAT) or behind a firewall, as long as the NAT or firewall allows the required outbound connections to the domain destinations in the preceding table. For connections going inbound to the Hybrid Data Security nodes, no ports should be visible from the internet. Within your data center, clients need access to the Hybrid Data Security nodes on TCP ports 443 and 22, for administrative purposes.

The URLs for the Common Identity (CI) hosts are region-specific. These are the current CI hosts:

Region	Common Identity Host URLs
Americas	https://idbroker.webex.com https://identity.webex.com https://idbroker-b-us.webex.com https://identity-b-us.webex.com
European Union	https://idbroker-eu.webex.com https://identity-eu.webex.com
Canada	https://idbroker-ca.webex.com https://identity-ca.webex.com
Singapore	https://idbroker-sg.webex.com https://identity-sg.webex.com
United Arab Emirates	https://idbroker-ae.webex.com https://identity-ae.webex.com

Proxy Server Requirements

We officially support the following proxy solutions that can integrate with your Hybrid Data Security nodes.
- Transparent proxy—Cisco Web Security Appliance (WSA).
- Explicit proxy—Squid.
  
  Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:) connections. To work around this issue, see Configure Squid Proxies for Hybrid Data Security.
We support the following authentication type combinations for explicit proxies:
- No authentication with HTTP or HTTPS
- Basic authentication with HTTP or HTTPS
- Digest authentication with HTTPS only
For a transparent inspecting proxy or an HTTPS explicit proxy, you must have a copy of the proxy's root certificate. The deployment instructions in this guide tell you how to upload the copy to the Hybrid Data Security nodes' trust stores.
The network hosting the HDS nodes must be configured to force outbound TCP traffic on port 443 to route through the proxy.
Proxies that inspect web traffic may interfere with web socket connections. If this problem occurs, bypassing (not inspecting) traffic to wbx2.com and ciscospark.com will solve the problem.

Complete the Prerequisites for Hybrid Data Security

Use this checklist to ensure that you are ready to install and configure your Hybrid Data Security cluster.

1	Make sure your partner organization has the Multi-Tenant HDS feature enabled and get the credentials of an account with partner full administrator and full administrator rights. Ensure your Webex customer organization is enabled for Pro Pack for Cisco Webex Control Hub. Contact your Cisco partner or account manager for help with this process. Customer organizations should not have any existing HDS deployment.
2	Choose a domain name for your HDS deployment (for example, `hds.company.com`) and obtain a certificate chain containing an X.509 certificate, private key, and any intermediate certificates. The certificate chain must meet the requirements in X.509 Certificate Requirements.
3	Prepare identical virtual hosts that you will set up as Hybrid Data Security nodes in your cluster. You need at least two separate hosts (3 recommended) colocated in the same secure data center, which meet the requirements in Virtual Host Requirements.
4	Prepare the database server that will act as the key data store for the cluster, according to the Database server requirements. The database server must be colocated in the secure data center with the virtual hosts. Create a database for key storage. (You must create this database—do not use the default database. The HDS applications, when installed, create the database schema.) Gather the details that the nodes will use to communicate with the database server: the host name or IP address (host) and port the name of the database (dbname) for key storage the username and password of a user with all privileges on the key storage database
5	For quick disaster recovery, set up a backup environment in a different data center. The backup environment mirrors the production environment of VMs and a backup database server. For example, if production has 3 VMs running HDS nodes, the backup environment should have 3 VMs.
6	Set up a syslog host to collect logs from the nodes in the cluster. Gather its network address and syslog port (default is UDP 514).
7	Create a secure backup policy for the Hybrid Data Security nodes, the database server, and the syslog host. At a minimum, to prevent unrecoverable data loss, you must back up the database and the configuration ISO file generated for the Hybrid Data Security nodes. Because the Hybrid Data Security nodes store the keys used in encryption and decryption of content, failure to maintain an operational deployment will result in the UNRECOVERABLE LOSS of that content. Webex App clients cache their keys, so an outage may not be immediately noticeable but will become evident over time. While temporary outages are impossible to prevent, they are recoverable. However, complete loss (no backups available) of either the database or configuration ISO file will result in unrecoverable customer data. The operators of the Hybrid Data Security nodes are expected to maintain frequent backups of the database and the configuration ISO file, and be prepared to rebuild the Hybrid Data Security data center if a catastrophic failure occurs.
8	Ensure that your firewall configuration allows connectivity for your Hybrid Data Security nodes as outlined in External connectivity requirements.
9	Install Docker ( https://www.docker.com) on any local machine running a supported OS (Microsoft Windows 10 Professional or Enterprise 64-bit, or Mac OSX Yosemite 10.10.3 or above) with a web browser that can access it at http://127.0.0.1:8080. You use the Docker instance to download and run the HDS Setup Tool, which builds the local configuration information for all the Hybrid Data Security nodes. You might need a Docker Desktop license. See Docker Desktop Requirements for more information. To install and run the HDS Setup Tool, the local machine must have the connectivity outlined in External connectivity requirements.
10	If you're integrating a proxy with Hybrid Data Security, make sure that it meets the Proxy Server Requirements.

Set up a Hybrid Data Security cluster

Hybrid Data Security Deployment Task Flow

Before you begin

1	Perform initial set up and download installation files Download the OVA file to your local machine for later use.
2	Create a Configuration ISO for the HDS Hosts Use the HDS Setup Tool to create an ISO configuration file for the Hybrid Data Security nodes.
3	Install the HDS Host OVA Create a virtual machine from the OVA file and perform initial configuration, such as network settings. The option to configure network settings during OVA deployment has been tested with ESXi 7.0. The option may not be available in earlier versions.
4	Set up the Hybrid Data Security VM Sign in to the VM console and set the sign-in credentials. Configure the network settings for the node if you didn't configure them at the time of OVA deployment.
5	Upload and Mount the HDS Configuration ISO Configure the VM from the ISO configuration file that you created with the HDS Setup Tool.
6	Configure the HDS Node for Proxy Integration If the network environment requires proxy configuration, specify the type of proxy that you will use for the node, and add the proxy certificate to the trust store if needed.
7	Register the first node in the cluster Register the VM with the Cisco Webex cloud as a Hybrid Data Security node.
8	Create and register more nodes Complete the cluster setup.
9	Activate Multi-Tenant HDS on Partner Hub. Activate HDS and manage tenant organizations on Partner Hub.

Perform initial set up and download installation files

In this task, you download an OVA file to your machine (not to the servers you set up as Hybrid Data Security nodes). You use this file later in the installation process.

1	Sign in to Partner Hub, and then click Services.
2	In the Cloud Services section, find the Hybrid Data Security card, and then click Set up. Clicking on Set up in Partner Hub is critical to the deployment process. Do not proceed with installation without completing this step.
3	Click Add a resource and click Download .OVA file on the Install and Configure Software card. Older versions of the software package (OVA) will not be compatible with the latest Hybrid Data Security upgrades. This can result in issues while upgrading the application. Make sure you download the latest version of the OVA file. You can also download the OVA at any time from the Help section. Click Settings > Help > Download Hybrid Data Security software. The OVA file automatically begins to download. Save the file to a location on your machine.
4	Optionally, click See hybrid data security deployment guide to check if there’s a later version of this guide available.

Create a Configuration ISO for the HDS Hosts

The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.

Before you begin

The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Partner Hub account with full administrator rights.

If you do not have a Docker Desktop license, you can use Podman Desktop to run the HDS Setup tool for steps 1 through 5 in the procedure below. See Run HDS Setup tool using Podman Desktop for details.

If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5 below. This table gives some possible environment variables:

Description	Variable
HTTP Proxy without authentication	`GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT`
HTTPS Proxy without authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT`
HTTP Proxy with authentication	`GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`
HTTPS Proxy with authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`

The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:
- Database credentials
- Certificate updates
- Changes to authorization policy
If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.

At your machine's command line, enter the appropriate command for your environment:

In regular environments:

docker rmi ciscocitg/hds-setup:stable

In FedRAMP environments:

docker rmi ciscocitg/hds-setup-fedramp:stable

This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.

To sign in to the Docker image registry, enter the following:

docker login -u hdscustomersro

At the password prompt, enter this hash:

dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo

Download the latest stable image for your environment:

In regular environments:

docker pull ciscocitg/hds-setup:stable

In FedRAMP environments:

docker pull ciscocitg/hds-setup-fedramp:stable

When the pull completes, enter the appropriate command for your environment:

In regular environments without a proxy:

docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable

In regular environments with an HTTP proxy:

docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable

In regular environments with an HTTPS proxy:

docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable

In FedRAMP environments without a proxy:

docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable

In FedRAMP environments with an HTTP proxy:

docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable

In FedRAMP environments with an HTTPS proxy:

docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable

When the container is running, you see "Express server listening on port 8080."

The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost.

Use a web browser to go to the localhost, http://127.0.0.1:8080, and enter admin username for Partner Hub at the prompt.

The tool uses this first entry of the username to set the proper environment for that account. The tool then displays the standard sign-in prompt.

When prompted, enter your Partner Hub admin sign-in credentials, and then click Log in to allow access to the required services for Hybrid Data Security.

On the Setup Tool overview page, click Get Started.

On the ISO Import page, you have these options:

No—If you’re creating your first HDS node, you don't have an ISO file to upload.
Yes—If you already created HDS nodes, then you select your ISO file in the browse and upload it.

Check that your X.509 certificate meets the requirements in X.509 Certificate Requirements.

If you never uploaded a certificate before, upload the X.509 certificate, enter the password, and click Continue.
If your certificate is OK, click Continue.
If your certificate has expired or you want to replace it, select No for Continue using HDS certificate chain and private key from previous ISO?. Upload a new X.509 certificate, enter the password, and click Continue.

Enter the database address and account for HDS to access your key datastore:

Select your Database Type (PostgreSQL or Microsoft SQL Server).

If you choose Microsoft SQL Server, you get an Authentication Type field.
(Microsoft SQL Server only) Select your Authentication Type:
- Basic Authentication: You need a local SQL Server account name in the Username field.
- Windows Authentication: You need a Windows account in the format username@DOMAIN in the Username field.
Enter the database server address in the form <hostname>:<port> or <IP-address>:<port>.

Example:
dbhost.example.org:1433 or 198.51.100.17:1433

You can use an IP address for basic authentication, if the nodes can't use DNS to resolve the hostname.

If you are using Windows authentication, you must enter a Fully Qualified Domain Name in the format dbhost.example.org:1433
Enter the Database Name.
Enter the Username and Password of a user with all privileges on the key storage database.

Select a TLS Database Connection Mode:

Mode	Description
Prefer TLS (default option)	HDS nodes don’t require TLS to connect to the database server. If you enable TLS on the database server, the nodes attempt an encrypted connection.
Require TLS	HDS nodes connect only if the database server can negotiate TLS.
Require TLS and verify certificate signer	This mode isn’t applicable for SQL Server databases. HDS nodes connect only if the database server can negotiate TLS. After establishing a TLS connection, the node compares the signer of the certificate from the database server to the certificate authority in the Database root certificate. If they don't match, the node drops the connection. Use the Database root certificate control below the drop-down to upload the root certificate for this option.
Require TLS and verify certificate signer and hostname	HDS nodes connect only if the database server can negotiate TLS. After establishing a TLS connection, the node compares the signer of the certificate from the database server to the certificate authority in the Database root certificate. If they don't match, the node drops the connection. The nodes also verify that the hostname in the server certificate matches the hostname in the Database host and port field. The names must match exactly, or the node drops the connection. Use the Database root certificate control below the drop-down to upload the root certificate for this option.

When you upload the root certificate (if necessary) and click Continue, the HDS Setup Tool tests the TLS connection to the database server. The tool also verifies the certificate signer and hostname, if applicable. If a test fails, the tool shows an error message describing the problem. You can choose whether to ignore the error and continue with the setup. (Because of connectivity differences, the HDS nodes might be able to establish the TLS connection even if the HDS Setup Tool machine can't successfully test it.)

On the System Logs page, configure your Syslogd server:

Enter the syslog server URL.

If the server isn’t DNS-resolvable from the nodes for your HDS cluster, use an IP address in the URL.

Example:
udp://10.92.43.23:514 indicates logging to Syslogd host 10.92.43.23 on UDP port 514.
If you set up your server to use TLS encryption, check Is your syslog server configured for SSL encryption?.

If you check this check box, make sure you enter a TCP URL such as tcp://10.92.43.23:514.
From the Choose syslog record termination drop-down, choose the appropriate setting for your ISO file: Choose or Newline is used for Graylog and Rsyslog TCP
- Null byte -- \x00
- Newline -- \n—Select this choice for Graylog and Rsyslog TCP.
Click Continue.

(Optional) You can change the default value for some database connection parameters in Advanced Settings. Generally, this parameter is the only one that you might want to change:

app_datasource_connection_pool_maxSize: 10

Click Continue on the Reset Service Accounts Password screen.

Service account passwords have a nine-month lifespan. Use this screen when your passwords are nearing expiry or you want to reset them to invalidate previous ISO files.

Click Download ISO File. Save the file in a location that's easy to find.

Make a backup copy of the ISO file on your local system.

Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes.

To shut down the Setup tool, type CTRL+C.

What to do next

Back up the configuration ISO file. You need it to create more nodes for recovery, or to make configuration changes. If you lose all copies of the ISO file, you've also lost the master key. Recovering the keys from your PostgreSQL or Microsoft SQL Server database isn't possible.

We never have a copy of this key and can't help if you lose it.

Install the HDS Host OVA

Use this procedure to create a virtual machine from the OVA file.

1	Use the VMware vSphere client on your computer to log into the ESXi virtual host.
2	Select File > Deploy OVF Template.
3	In the wizard, specify the location of the OVA file that you downloaded earlier, and then click Next.
4	On the Select a name and folder page, enter a Virtual machine name for the node (for example, "HDS_Node_1"), choose a location where the virtual machine node deployment can reside, and then click Next.
5	On the Select a compute resource page, choose the destination compute resource, and then click Next. A validation check runs. After it finishes, the template details appear.
6	Verify the template details and then click Next.
7	If you are asked to choose the resource configuration on the Configuration page, click 4 CPU and then click Next.
8	On the Select storage page, click Next to accept the default disk format and VM storage policy.
9	On the Select networks page, choose the network option from the list of entries to provide the desired connectivity to the VM.
10	On the Customize template page, configure the following network settings: Hostname—Enter the FQDN (hostname and domain) or a single word hostname for the node. You do not need to set the domain to match the domain that you used to obtain the X.509 certificate. To ensure a successful registration to the cloud, use only lowercase characters in the FQDN or hostname that you set for the node. Capitalization is not supported at this time. The total length of the FQDN must not exceed 64 characters. IP Address— Enter the IP address for the internal interface of the node. Your node should have an internal IP address and DNS name. DHCP is not supported. Mask—Enter the subnet mask address in dot-decimal notation. For example, 255.255.255.0. Gateway—Enter the gateway IP address. A gateway is a network node that serves as an access point to another network. DNS Servers—Enter a comma-separated list of DNS servers, which handle translating domain names to numeric IP addresses. (Up to 4 DNS entries are allowed.) NTP Servers—Enter your organization's NTP server or another external NTP server that can be used in your organization. The default NTP servers may not work for all enterprises. You can also use a comma-separated list to enter multiple NTP servers. Deploy all the nodes on the same subnet or VLAN, so that all nodes in a cluster are reachable from clients in your network for administrative purposes. If preferred, you can skip the network setting configuration and follow the steps in Set up the Hybrid Data Security VM to configure the settings from the node console. The option to configure network settings during OVA deployment has been tested with ESXi 7.0. The option may not be available in earlier versions.
11	Right-click the node VM, and then choose Power > Power On. The Hybrid Data Security software is installed as a guest on the VM Host. You are now ready to sign in to the console and configure the node. Troubleshooting Tips You may experience a delay of a few minutes before the node containers come up. A bridge firewall message appears on the console during first boot, during which you can't sign in.

Set up the Hybrid Data Security VM

Use this procedure to sign in to the Hybrid Data Security node VM console for the first time and set the sign-in credentials. You can also use the console to configure the network settings for the node if you didn't configure them at the time of OVA deployment.

1	In the VMware vSphere client, select your Hybrid Data Security node VM and select the Console tab. The VM boots up and a login prompt appears. If the login prompt does not display, press Enter.
2	Use the following default login and password to sign in and change the credentials: Login: `admin` Password: `cisco` Since you are signing in to your VM for the first time, you are required to change the administrator password.
3	If you already configured the network settings in Install the HDS Host OVA, skip the rest of this procedure. Otherwise, in the main menu, select the Edit Configuration option.
4	Set up a static configuration with IP address, Mask, Gateway and DNS information. Your node should have an internal IP address and DNS name. DHCP is not supported.
5	(Optional) Change the hostname, domain or NTP server(s), if needed to match your network policy. You do not need to set the domain to match the domain that you used to obtain the X.509 certificate.
6	Save the network configuration and reboot the VM so that the changes take effect.

Upload and Mount the HDS Configuration ISO

Use this procedure to configure the virtual machine from the ISO file that you created with the HDS Setup Tool.

Before you begin

Because the ISO file holds the master key, it should only be exposed on a "need to know" basis, for access by the Hybrid Data Security VMs and any administrators who might need to make changes. Make sure that only those administrators can access the datastore.

Upload the ISO file from your computer:

In the VMware vSphere client's left navigation pane, click on the ESXi server.
On the Configuration tab's Hardware list, click Storage.
In the Datastores list, right-click on the datastore for your VMs and click Browse Datastore.
Click on the Upload Files icon, and then click Upload File.
Browse to the location where you downloaded the ISO file on your computer and click Open.
Click Yes to accept the upload/download operation warning, and close the datastore dialog.

Mount the ISO file:

In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.
Click OK to accept the restricted edit options warning.
Click CD/DVD Drive 1, select the option to mount from a datastore ISO file, and browse to the location where you uploaded the configuration ISO file.
Check Connected and Connect at power on.
Save your changes and reboot the virtual machine.

What to do next

If your IT policy requires, you can optionally unmount the ISO file after all your nodes pick up the configuration changes. See (Optional) Unmount ISO After HDS Configuration for details.

Configure the HDS Node for Proxy Integration

If the network environment requires a proxy, use this procedure to specify the type of proxy that you want to integrate with Hybrid Data Security. If you choose a transparent inspecting proxy or an HTTPS explicit proxy, you can use the node's interface to upload and install the root certificate. You can also check the proxy connection from the interface, and troubleshoot any potential issues.

Before you begin

See Proxy Support for an overview of the supported proxy options.
Proxy Server Requirements

1	Enter the HDS node setup URL `https://[HDS Node IP or FQDN]/setup` in a web browser, enter the admin credentials that you set up for the node, and then click Sign In.
2	Go to Trust Store & Proxy, and then choose an option: No Proxy—The default option before you integrate a proxy. No certificate update is required. Transparent Non-Inspecting Proxy—Nodes are not configured to use a specific proxy server address and should not require any changes to work with a non-inspecting proxy. No certificate update is required. Transparent Inspecting Proxy—Nodes are not configured to use a specific proxy server address. No HTTPS configuration changes are necessary on the Hybrid Data Security deployment, however, the HDS nodes need a root certificate so that they trust the proxy. Inspecting proxies are typically used by IT to enforce policies on which websites can be visited and which types of content are not permitted. This type of proxy decrypts all your traffic (even HTTPS). Explicit Proxy—With explicit proxy, you tell the client (HDS nodes) which proxy server to use, and this option supports several authentication types. After you choose this option, you must enter the following information: Proxy IP/FQDN—Address that can be used to reach the proxy machine. Proxy Port—A port number that the proxy uses to listen for proxied traffic. Proxy Protocol—Choose http (views and controls all requests that are received from the client) or https (provides a channel to the server and the client receives and validates the server's certificate). Choose an option based on what your proxy server supports. Authentication Type—Choose from among the following authentication types: None—No further authentication is required. Available for HTTP or HTTPS proxies. Basic—Used for an HTTP User Agent to provide a user name and password when making a request. Uses Base64 encoding. Available for HTTP or HTTPS proxies. If you choose this option, you must also enter the user name and password. Digest—Used to confirm the account before sending sensitive information. Applies a hash function on the user name and password before sending over the network. Available for HTTPS proxies only. If you choose this option, you must also enter the user name and password. Follow the next steps for a transparent inspecting proxy, an HTTP explicit proxy with Basic authentication, or an HTTPS explicit proxy.
3	Click Upload a Root Certificate or End Entity Certificate, and then navigate to a choose the root certificate for the proxy. The certificate is uploaded but not yet installed because you must reboot the node to install the certificate. Click the chevron arrow by the certificate issuer name to get more details or click Delete if you made a mistake and want to reupload the file.
4	Click Check Proxy Connection to test the network connectivity between the node and the proxy. If the connection test fails, you'll see an error message that shows the reason and how you can correct the issue. If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server. This condition is expected in many explicit proxy configurations. You can continue with the setup, and the node will function in Blocked External DNS Resolution mode. If you think this is an error, complete these steps, and then see Turn off Blocked External DNS Resolution Mode.
5	After the connection test passes, for explicit proxy set to https only, turn the toggle on to Route all port 443/444 https requests from this node through the explicit proxy. This setting requires 15 seconds to take effect.
6	Click Install All Certificates Into the Trust Store (appears for an HTTPS explicit proxy or a transparent inspecting proxy) or Reboot (appears for an HTTP explicit proxy), read the prompt, and then click Install if you're ready. The node reboots within a few minutes.
7	After the node reboots, sign in again if needed, and then open the Overview page to check the connectivity checks to make sure they are all in green status. The proxy connection check only tests a subdomain of webex.com. If there are connectivity problems, a common issue is that some of the cloud domains listed in the install instructions are being blocked at the proxy.

Register the first node in the cluster

This task takes the generic node that you created in the Set up the Hybrid Data Security VM, registers the node with the Webex cloud, and turns it into a Hybrid Data Security node.

When you register your first node, you create a cluster to which the node is assigned. A cluster contains one or more nodes deployed to provide redundancy.

Before you begin

Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.
Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.

1	Sign in to https://admin.webex.com.
2	From the menu on the left side of the screen, select Services.
3	In the Cloud Services section, find Hybrid Data Security card and click Set up.
4	In the page that opens, click Add a resource.
5	In the first field of Add a node card, enter a name for the cluster to which you want to assign your Hybrid Data Security node. We recommend that you name a cluster based on where the nodes of the cluster are located geographically. Examples: "San Francisco" or "New York" or "Dallas"
6	In the second field, enter the internal IP address or fully qualified domain name (FQDN) of your node and click Add at the bottom of the screen. This IP address or FQDN should match the IP address or hostname and domain that you used in Set up the Hybrid Data Security VM. A message appears indicating you can register your node to the Webex.
7	Click Go to Node. After a few moments, you are redirected to the node connectivity tests for Webex services. If all tests are successful, the Allow Access to Hybrid Data Security Node page appears. There, you confirm that you want to give permissions to your Webex organization to access your node.
8	Check the Allow Access to Your Hybrid Data Security Node checkbox, and then click Continue. Your account is validated and the "Registration Complete" message indicates that your node is now registered to the Webex cloud.
9	Click the link or close the tab to go back to the Partner Hub Hybrid Data Security page. On the Hybrid Data Security page, the new cluster containing the node that you registered is displayed under the Resources tab. The node will automatically download the latest software from the cloud.

Create and register more nodes

To add additional nodes to your cluster, you simply create additional VMs and mount the same configuration ISO file, then register the node. We recommend that you have at least 3 nodes.

Before you begin

Once you begin registration of a node, you must complete it within 60 minutes or you have to start over.
Ensure that any pop-up blockers in your browser are disabled or that you allow an exception for admin.webex.com.

1	Create a new virtual machine from the OVA, repeating the steps in Install the HDS Host OVA.
2	Set up the initial configuration on the new VM, repeating the steps in Set up the Hybrid Data Security VM.
3	On the new VM, repeat the steps in Upload and Mount the HDS Configuration ISO.
4	If you are setting up a proxy for your deployment, repeat the steps in Configure the HDS Node for Proxy Integration as needed for the new node.
5	Register the node. In https://admin.webex.com, select Services from the menu on the left side of the screen. In the Cloud Services section, find the Hybrid Data Security card and click View all. The Hybrid Data Security Resources page appears. The newly created cluster will appear in the Resources page. Click on the cluster to view the nodes assigned to the cluster. Click Add a node on the right side of the screen. Enter the internal IP address or fully qualified domain name (FQDN) of your node and click Add. A page opens with a message indicating you can register your node to the Webex cloud. After a few moments, you are redirected to the node connectivity tests for Webex services. If all tests are successful, the Allow Access to Hybrid Data Security Node page appears. There, you confirm that you want to give permissions to your organization to access your node. Check the Allow Access to Your Hybrid Data Security Node checkbox, and then click Continue. Your account is validated and the "Registration Complete" message indicates that your node is now registered to the Webex cloud. Click the link or close the tab to go back to the Partner Hub Hybrid Data Security page. Node added pop up message also appears at the bottom of the screen in Partner Hub. Your node is registered.

Manage Tenant organizations on Multi-Tenant Hybrid Data Security

Activate Multi-Tenant HDS on Partner Hub

This task ensures all users of the customer organizations can start leveraging HDS for On-Premises encryption keys and other security services.

Before you begin

Ensure you have completed setting up your Multi-Tenant HDS cluster with the required number of nodes.

1	Sign in to https://admin.webex.com.
2	From the menu on the left side of the screen, select Services.
3	In the Cloud Services section, find Hybrid Data Security and click Edit Settings.
4	Click Activate HDS on the HDS Status card.

Add tenant organizations in Partner Hub

In this task, you assign customer organizations to your Hybrid Data Security Cluster.

1	Sign in to https://admin.webex.com.
2	From the menu on the left side of the screen, select Services.
3	In the Cloud Services section, find Hybrid Data Security and click View all.
4	Click on the cluster to which you want a customer to be assigned.
5	Go to the Assigned customers tab.
6	Click Add customers.
7	Select the customer you want to add from the drop down menu.
8	Click Add, the customer will be added to the cluster.
9	Repeat steps 6 to 8 to add multiple customers to your cluster.
10	Click Done on the bottom of the screen once you have added the customers.

What to do next

Run the HDS Setup tool as detailed in Create Customer Main Keys (CMKs) using the HDS Setup tool to complete the setup process.

Create Customer Main Keys (CMKs) using the HDS Setup tool

Before you begin

Assign customers to the appropriate cluster as detailed in Add tenant organizations in Partner Hub. Run the HDS Setup tool to complete the set up process for the newly added customer organizations.

If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5. This table gives some possible environment variables:

Description	Variable
HTTP Proxy without authentication	`GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT`
HTTPS Proxy without authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT`
HTTP Proxy with authentication	`GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`
HTTPS Proxy with authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`

The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:
- Database credentials
- Certificate updates
- Changes to authorization policy
If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.

The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.

1	At your machine's command line, enter the appropriate command for your environment: In regular environments: `docker rmi ciscocitg/hds-setup:stable` In FedRAMP environments: `docker rmi ciscocitg/hds-setup-fedramp:stable` This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.
2	To sign in to the Docker image registry, enter the following: `docker login -u hdscustomersro`
3	At the password prompt, enter this hash: `dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo`
4	Download the latest stable image for your environment: In regular environments: `docker pull ciscocitg/hds-setup:stable` In FedRAMP environments: `docker pull ciscocitg/hds-setup-fedramp:stable`
5	When the pull completes, enter the appropriate command for your environment: In regular environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable` In regular environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In regular environments with an HTTPS proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In FedRAMP environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTPS proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` When the container is running, you see "Express server listening on port 8080."
6	The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost. Use a web browser to go to the localhost, `http://127.0.0.1:8080`, and enter admin username for Partner Hub at the prompt. The tool uses this first entry of the username to set the proper environment for that account. The tool then displays the standard sign-in prompt.
7	When prompted, enter your Partner Hub admin sign-in credentials, and then click Log in to allow access to the required services for Hybrid Data Security.
8	On the Setup Tool overview page, click Get Started.
9	On the ISO Import page, click Yes.
10	Select your ISO file in the browser and upload it. Ensure connectivity to your database to perform CMK management.
11	Go to the Tenant CMK Management tab, where you will find the following three ways to manage Tenant CMKs. Create CMK for all ORGs or Create CMK - Click on this button on the banner at the top of the screen to create CMKs for all newly added organizations. Click Manage CMKs button on the right side of the screen and click Create CMKs to create CMKs for all newly added organizations. Click … near the CMK management pending status of a specific organization in the table and click Create CMK to create CMK for that organization.
12	Once CMK creation is successful, the status in the table will change from CMK management pending to CMK managed.
13	If CMK creation is unsuccessful, an error will be displayed.

Remove tenant organizations

Before you begin

Once removed, users of customer organizations will not be able to leverage HDS for their encryption needs and will lose all existing spaces. Before removing customer organizations, please contact your Cisco partner or account manager.

1	Sign in to https://admin.webex.com.
2	From the menu on the left side of the screen, select Services.
3	In the Cloud Services section, find Hybrid Data Security and click View all.
4	On the Resources tab, click on the cluster from which you would like to remove customer organizations.
5	In the page that opens, click Assigned Customers.
6	From the list of customer organizations that are displayed, click ... on the right side of the customer organization you would like to remove and click Remove from cluster.

What to do next

Complete the removal process by revoking the CMKs of the customer organizations as detailed in Revoke CMKs of tenants removed from HDS.

Revoke CMKs of tenants removed from HDS.

Before you begin

Remove customers from the appropriate cluster as detailed in Remove tenant organizations. Run the HDS Setup tool to complete the removal process for the customer organizations that were removed.

If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5. This table gives some possible environment variables:

Description	Variable
HTTP Proxy without authentication	`GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT`
HTTPS Proxy without authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT`
HTTP Proxy with authentication	`GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`
HTTPS Proxy with authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`

The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:
- Database credentials
- Certificate updates
- Changes to authorization policy
If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.

The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.

1	At your machine's command line, enter the appropriate command for your environment: In regular environments: `docker rmi ciscocitg/hds-setup:stable` In FedRAMP environments: `docker rmi ciscocitg/hds-setup-fedramp:stable` This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.
2	To sign in to the Docker image registry, enter the following: `docker login -u hdscustomersro`
3	At the password prompt, enter this hash: `dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo`
4	Download the latest stable image for your environment: In regular environments: `docker pull ciscocitg/hds-setup:stable` In FedRAMP environments: `docker pull ciscocitg/hds-setup-fedramp:stable`
5	When the pull completes, enter the appropriate command for your environment: In regular environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable` In regular environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In regular environments with an HTTPS proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In FedRAMP environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTPS proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` When the container is running, you see "Express server listening on port 8080."
6	The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost. Use a web browser to go to the localhost, `http://127.0.0.1:8080`, and enter admin username for Partner Hub at the prompt. The tool uses this first entry of the username to set the proper environment for that account. The tool then displays the standard sign-in prompt.
7	When prompted, enter your Partner Hub admin sign-in credentials, and then click Log in to allow access to the required services for Hybrid Data Security.
8	On the Setup Tool overview page, click Get Started.
9	On the ISO Import page, click Yes.
10	Select your ISO file in the browser and upload it.
11	Go to the Tenant CMK Management tab, where you will find the following three ways to manage Tenant CMKs. Revoke CMK for all ORGs or Revoke CMK - Click on this button on the banner at the top of the screen to revoke CMKs of all organizations that were removed. Click Manage CMKs button on the right side of the screen and click Revoke CMKs to revoke CMKs of all organizations that were removed. Click … near the CMK to be revoked status of a specific organization in the table and click Revoke CMK to revoke CMK for that specific organization.
12	Once CMK revocation is successful, the customer organization will no longer appear in the table.
13	If CMK revocation is unsuccessful, an error will be displayed.

Test your Hybrid Data Security deployment

Test Your Hybrid Data Security Deployment

Use this procedure to test Multi-Tenant Hybrid Data Security encryption scenarios.

Before you begin

Set up your Multi-Tenant Hybrid Data Security deployment.
Ensure that you have access to the syslog to verify that key requests are passing to your Multi-Tenant Hybrid Data Security deployment.

Keys for a given space are set by the creator of the space. Sign in to the Webex App as one of the customer organization users, and then create a space.

If you deactivate the Hybrid Data Security deployment, content in spaces that users create is no longer accessible once the client-cached copies of the encryption keys are replaced.

Send messages to the new space.

Check the syslog output to verify that the key requests are passing to your Hybrid Data Security deployment.

To check for a user first establishing a secure channel to the KMS, filter on kms.data.method=create and kms.data.type=EPHEMERAL_KEY_COLLECTION:

You should find an entry such as the following (identifiers shortened for readability):

2020-07-21 17:35:34.562 (+0000) INFO  KMS [pool-14-thread-1] - [KMS:REQUEST] received, 
deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/0[~]9 ecdheKid: kms://hds2.org5.portun.us/statickeys/3[~]0 
(EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_d[~]0, kms.data.method=create, 
kms.merc.id=8[~]a, kms.merc.sync=false, kms.data.uriHost=hds2.org5.portun.us, kms.data.type=EPHEMERAL_KEY_COLLECTION, 
kms.data.requestId=9[~]6, kms.data.uri=kms://hds2.org5.portun.us/ecdhe, kms.data.userId=0[~]2

To check for a user requesting an existing key from the KMS, filter on kms.data.method=retrieve and kms.data.type=KEY:

You should find an entry such as:

2020-07-21 17:44:19.889 (+0000) INFO  KMS [pool-14-thread-31] - [KMS:REQUEST] received, 
deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
(EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_f[~]0, kms.data.method=retrieve, 
kms.merc.id=c[~]7, kms.merc.sync=false, kms.data.uriHost=ciscospark.com, kms.data.type=KEY, 
kms.data.requestId=9[~]3, kms.data.uri=kms://ciscospark.com/keys/d[~]2, kms.data.userId=1[~]b

To check for a user requesting the creation of a new KMS key, filter on kms.data.method=create and kms.data.type=KEY_COLLECTION:

You should find an entry such as:

2020-07-21 17:44:21.975 (+0000) INFO  KMS [pool-14-thread-33] - [KMS:REQUEST] received, 
deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
(EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_4[~]0, kms.data.method=create, 
kms.merc.id=6[~]e, kms.merc.sync=false, kms.data.uriHost=null, kms.data.type=KEY_COLLECTION, 
kms.data.requestId=6[~]4, kms.data.uri=/keys, kms.data.userId=1[~]b

To check for a user requesting the creation of a new KMS Resource Object (KRO) when a space or other protected resource is created, filter on kms.data.method=create and kms.data.type=RESOURCE_COLLECTION:

You should find an entry such as:

2020-07-21 17:44:22.808 (+0000) INFO  KMS [pool-15-thread-1] - [KMS:REQUEST] received, 
deviceId: https://wdm-a.wbx2.com/wdm/api/v1/devices/f[~]f ecdheKid: kms://hds2.org5.portun.us/ecdhe/5[~]1 
(EncryptionKmsMessageHandler.java:312) WEBEX_TRACKINGID=HdsIntTest_d[~]0, kms.data.method=create, 
kms.merc.id=5[~]3, kms.merc.sync=true, kms.data.uriHost=null, kms.data.type=RESOURCE_COLLECTION, 
kms.data.requestId=d[~]e, kms.data.uri=/resources, kms.data.userId=1[~]b

Monitor Hybrid Data Security Health

A status indicator within Partner Hub shows you whether all is well with the Multi-Tenant Hybrid Data Security deployment. For more proactive alerting, sign up for email notifications. You'll be notified when there are service-impacting alarms or software upgrades.

1	In Partner Hub, select Services from the menu on the left side of the screen.
2	In the Cloud Services section, find Hybrid Data Security and click Edit Settings. The Hybrid Data Security Settings page appears.
3	In the Email Notifications section, type one or more email addresses separated by commas, and press Enter.

Manage your HDS deployment

Manage HDS Deployment

Use the tasks described here to manage your Hybrid Data Security deployment.

Set Cluster Upgrade Schedule

Software upgrades for Hybrid Data Security are done automatically at the cluster level, which ensures that all nodes are always running the same software version. Upgrades are done according to the upgrade schedule for the cluster. When a software upgrade becomes available, you have the option of manually upgrading the cluster before the scheduled upgrade time. You can set a specific upgrade schedule or use the default schedule of 3:00 AM Daily United States: America/Los Angeles. You can also choose to postpone an upcoming upgrade, if necessary.

To set the upgrade schedule:

1	Sign in to Partner Hub.
2	From the menu on the left side of the screen, select Services.
3	In the Cloud Services section, find Hybrid Data Security and click Set up
4	On the Hybrid Data Security Resources page, select the cluster.
5	Click on the Cluster Settings tab.
6	On the Cluster Settings page, under Upgrade Schedule, select the time and time zone for the upgrade schedule. Note: Under the time zone, the next available upgrade date and time is displayed. You can postpone the upgrade to the following day, if needed, by clicking Postpone by 24 hours.

Change the Node Configuration

Occasionally you may need to change the configuration of your Hybrid Data Security node for a reason such as:

Changing x.509 certificates due to expiration or other reasons.

We don't support changing the CN domain name of a certificate. The domain must match the original domain used to register the cluster.
Updating database settings to change to a replica of the PostgreSQL or Microsoft SQL Server database.

We don’t support migrating data from PostgreSQL to Microsoft SQL Server, or the opposite way. To switch the database environment, start a new deployment of Hybrid Data Security.
Creating a new configuration to prepare a new data center.

Also, for security purposes, Hybrid Data Security uses service account passwords that have a nine-month lifespan. After the HDS Setup tool generates these passwords, you deploy them to each of your HDS nodes in the ISO config file. When your organization's passwords are nearing expiration, you receive a notice from the Webex team to reset the password for your machine account. (The email includes the text, "Use the machine account API to update the password.") If your passwords haven't expired yet, the tool gives you two options:

Soft reset—The old and new passwords both work for up to 10 days. Use this period to replace the ISO file on the nodes gradually.
Hard reset—The old passwords stop working immediately.

If your passwords expire without a reset, it impacts your HDS service, requiring an immediate hard reset and replacement of the ISO file on all nodes.

Use this procedure to generate a new configuration ISO file and apply it to your cluster.

Before you begin

The HDS Setup tool runs as a Docker container on a local machine. To access it, run Docker on that machine. The setup process requires the credentials of a Partner Hub account with partner full administrator rights .

If you do not have a Docker Desktop license, you can use Podman Desktop to run the HDS Setup tool for steps 1.a through 1.e in the procedure below. See Run HDS Setup tool using Podman Desktop for details.

If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in 1.e. This table gives some possible environment variables:

Description	Variable
HTTP Proxy without authentication	`GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT`
HTTPS Proxy without authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT`
HTTP Proxy with authentication	`GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`
HTTPS Proxy with authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`

You need a copy of the current configuration ISO file to generate a new configuration. The ISO contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the ISO when you make configuration changes, including database credentials, certificate updates, or changes to authorization policy.

1	Using Docker on a local machine, run the HDS Setup Tool. At your machine's command line, enter the appropriate command for your environment: In regular environments: `docker rmi ciscocitg/hds-setup:stable` In FedRAMP environments: `docker rmi ciscocitg/hds-setup-fedramp:stable` This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore. To sign in to the Docker image registry, enter the following: `docker login -u hdscustomersro` At the password prompt, enter this hash: `dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo` Download the latest stable image for your environment: In regular environments: `docker pull ciscocitg/hds-setup:stable` In FedRAMP environments: `docker pull ciscocitg/hds-setup-fedramp:stable` Make sure you pull the latest Setup tool for this procedure. Versions of the tool created before February 22, 2018 don’t have the password reset screens. When the pull completes, enter the appropriate command for your environment: In regular environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable` In regular environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In regular environments with an HTTPSproxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In FedRAMP environments without a proxy: `docker run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTP proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTPS proxy: `docker run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` When the container is running, you see "Express server listening on port 8080." Use a browser to connect to the localhost, `http://127.0.0.1:8080`. The Setup tool does not support connecting to localhost through http://localhost:8080. Use http://127.0.0.1:8080 to connect to localhost. When prompted, enter your Partner Hub customer sign-in credentials and then click Accept to continue. Import the current configuration ISO file. Follow the prompts to complete the tool and download the updated file. To shut down the Setup tool, type `CTRL+C`. Create a backup copy of the updated file in another data center.
2	If you only have one HDS node running, create a new Hybrid Data Security node VM and register it using the new configuration ISO file. For more detailed instructions, see Create and register more nodes. Install the HDS host OVA. Set up the HDS VM. Mount the updated configuration file. Register the new node in Partner Hub.
3	For existing HDS nodes that are running the older configuration file, mount the ISO file. Perform the following procedure on each node in turn, updating each node before turning off the next node: Turn off the virtual machine. In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings. Click `CD/DVD Drive 1`, select the option to mount from an ISO file, and browse to the location where you downloaded the new configuration ISO file. Check Connect at power on. Save your changes and power on the virtual machine.
4	Repeat step 3 to replace the configuration on each remaining node that is running the old configuration.

Turn off Blocked External DNS Resolution Mode

When you register a node or check the node's proxy configuration, the process tests DNS look-up and connectivity to the Cisco Webex cloud. If the node's DNS server can't resolve public DNS names, the node automatically goes into Blocked External DNS Resolution mode.

If your nodes are able to resolve public DNS names through internal DNS servers, you can turn off this mode by rerunning the proxy connection test on each node.

Before you begin

Ensure that your internal DNS servers can resolve public DNS names, and that your nodes can communicate with them.

1	In a web browser, open the Hybrid Data Security node interface (IP address/setup, for example, https://192.0.2.0/setup), enter the admin credentials you set up for the node, and then click Sign In.
2	Go to Overview (the default page). When enabled, Blocked External DNS Resolution is set to Yes.
3	Go to the Trust Store & Proxy page.
4	Click Check Proxy Connection. If you see a message saying that external DNS resolution was not successful, the node was unable to reach the DNS server and will remain in this mode. Otherwise, after you reboot the node and go back to the Overview page, Blocked External DNS Resolution should be set to no.

What to do next

Repeat the proxy connection test on each node in your Hybrid Data Security cluster.

Remove a Node

Use this procedure to remove a Hybrid Data Security node from the Webex cloud. After you remove the node from the cluster, delete the virtual machine to prevent further access to your security data.

Use the VMware vSphere client on your computer to log into the ESXi virtual host and power off the virtual machine.

Remove the node:

Sign in to Partner Hub, and then select Services.
On the Hybrid Data Security card, click View All to display the Hybrid Data Security Resources page.
Select your cluster to display its Overview panel.
Click on the node you want to remove.
Click Deregister this node on the panel that appears on the right
You can also deregister the node by clicking … on the right side of the node and selecting Remove this node.

In the vSphere client, delete the VM. (In the left navigation pane, right-click on the VM and click Delete.)

If you don’t delete the VM, remember to unmount the configuration ISO file. Without the ISO file, you can't use the VM to access your security data.

Disaster Recovery using Standby Data Center

The most critical service that your Hybrid Data Security cluster provides is the creation and storage of keys used to encrypt messages and other content stored in the Webex cloud. For each user within the organization who is assigned to Hybrid Data Security, new key creation requests are routed to the cluster. The cluster is also responsible for returning the keys that it's created to any users authorized to retrieve them, for example, members of a conversation space.

Because the cluster performs the critical function of providing these keys, it's imperative that the cluster remains running and that proper backups are maintained. Loss of the Hybrid Data Security database or of the configuration ISO used for the schema will result in UNRECOVERABLE LOSS of customer content. The following practices are mandatory to prevent such a loss:

If a disaster causes the HDS deployment in the primary data center to become unavailable, follow this procedure to manually failover to the standby data center.

Before you begin

De-register all nodes from Partner Hub as mentioned in Remove a Node. Use the latest ISO file that was configured against the nodes of the cluster that was previously active, to perform the failover procedure mentioned below.

1	Start the HDS Setup tool and follow the steps mentioned in Create a Configuration ISO for the HDS Hosts.
2	Complete the configuration process and save the ISO file in a location that's easy to find.
3	Make a backup copy of the ISO file on your local system. Keep the backup copy secure. This file contains a master encryption key for the database contents. Restrict access to only those Hybrid Data Security administrators who should make configuration changes.
4	In the VMware vSphere client's left navigation pane, right-click on the VM and click Edit Settings.
5	Click Edit Settings >CD/DVD Drive 1 and select Datastore ISO File. Make sure Connected and Connect at power on are checked so that updated configuration changes can take effect after starting the nodes.
6	Power on the HDS node and make sure there are no alarms for at least 15 minutes.
7	Register the node in Partner hub. Refer Register the first node in the cluster.
8	Repeat the process for every node in the standby data center.

What to do next

After failover, if the primary data center becomes active again, de-register the nodes of the standby data center and repeat the process of configuring ISO and registering nodes of the primary data center as mentioned above.

(Optional) Unmount ISO After HDS Configuration

The standard HDS configuration runs with the ISO mounted. But, some customers prefer not leaving ISO files continuously mounted. You can unmount the ISO file after all HDS nodes pick up the new configuration.

You still use the ISO files to make configuration changes. When you create a new ISO or update an ISO through the Setup Tool, you must mount the updated ISO on all your HDS nodes. Once all your nodes have picked up the configuration changes, you can unmount the ISO again with this procedure.

Before you begin

Upgrade all your HDS nodes to version 2021.01.22.4720 or later.

1	Shut down one of your HDS nodes.
2	In the vCenter Server Appliance, select the HDS node.
3	Choose Edit Settings > CD/DVD drive and uncheck Datastore ISO File.
4	Power on the HDS node and ensure there are no alarms for atleast 20 minutes.
5	Repeat for each HDS node in turn.

Troubleshoot Hybrid Data Security

View Alerts and Troubleshoot

A Hybrid Data Security deployment is considered unavailable if all nodes in the cluster are unreachable, or the cluster is working so slowly that requests time out. If users cannot reach your Hybrid Data Security cluster, they experience the following symptoms:

New spaces cannot be created (unable to create new keys)
Messages and space titles fail to decrypt for:
- New users added to a space (unable to fetch keys)
- Existing users in a space using a new client (unable to fetch keys)
Existing users in a space will continue to run successfully as long as their clients have a cache of the encryption keys

It's important that you properly monitor your Hybrid Data Security cluster and address any alerts promptly to avoid disruption of service.

Alerts

If there is a problem with the Hybrid Data Security setup, Partner Hub displays alerts to the organization administrator, and sends emails to the configured email address. The alerts cover many common scenarios.

Table 1. Common Issues and the Steps to Resolve Them
Alert	Action
Local database access failure.	Check for database errors or local network issues.
Local database connection failure.	Check that the database server is available, and the right service account credentials were used in node configuration.
Cloud service access failure.	Check that the nodes can access the Webex servers as specified in External connectivity requirements.
Renewing cloud service registration.	Registration to cloud services was dropped. Renewal of registration is in progress.
Cloud service registration dropped.	Registration to cloud services terminated. Service is shutting down.
Service not yet activated.	Activate HDS in Partner Hub.
Configured domain does not match server certificate.	Ensure that your server certificate matches the configured service activation domain. The most likely cause is that the certificate CN was recently changed and is now different from the CN that was used during initial setup.
Failed to authenticate to cloud services.	Check for accuracy and possible expiration of service account credentials.
Failed to open local keystore file.	Check for integrity and password accuracy on local keystore file.
Local server certificate is invalid.	Check the server certificate's expiration date and confirm that it was issued by a trusted Certificate Authority.
Unable to post metrics.	Check local network access to external cloud services.
/media/configdrive/hds directory does not exist.	Check the ISO mount configuration on virtual host. Verify that the ISO file exists, that it is configured to mount on reboot, and that it mounts successfully.
Tenant Org Setup is not completed for the added orgs	Complete setup by creating CMKs for newly added tenant organizations using HDS Setup Tool.
Tenant Org Setup is not completed for the removed orgs	Complete setup by revoking CMKs of tenant organizations that were removed using HDS Setup Tool.

Troubleshoot Hybrid Data Security

Use the following general guidelines when troubleshooting problems with Hybrid Data Security.

1	Review Partner Hub for any alerts and fix any items you find there. See the image below for reference.
2	Review the syslog server output for activity from the Hybrid Data Security deployment. Filter for words like "Warning" and "Error" to help in troubleshooting.
3	Contact Cisco support.

Other notes

Known Issues for Hybrid Data Security

If you shut down your Hybrid Data Security cluster (by deleting it in Partner Hub or by shutting down all nodes), lose your configuration ISO file, or lose access to the keystore database, Webex App users of customer organizations can no longer use spaces under their People list that were created with keys from your KMS. We do not currently have a workaround or fix for this issue and urge you not to shut down your HDS services once they are handling active user accounts.
A client which has an existing ECDH connection to a KMS maintains that connection for a period of time (likely one hour).

Run HDS Setup tool using Podman Desktop

Podman is a free and open-source container management tool that provides a way to run, manage and create containers. Podman Desktop can be downloaded from https://podman-desktop.io/downloads.

The HDS Setup tool runs as a Docker container on a local machine. To access it, download and run Podman on that machine. The setup process requires the credentials of a Control Hub account with full administrator rights for your organization.

If the HDS Setup tool runs behind a proxy in your environment, provide the proxy settings (server, port, credentials) through Docker environment variables when bringing up the Docker container in step 5. This table gives some possible environment variables:

Description	Variable
HTTP Proxy without authentication	`GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT`
HTTPS Proxy without authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT`
HTTP Proxy with authentication	`GLOBAL_AGENT_HTTP_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`
HTTPS Proxy with authentication	`GLOBAL_AGENT_HTTPS_PROXY=http://USERNAME:PASSWORD@SERVER_IP:PORT`

The configuration ISO file that you generate contains the master key encrypting the PostgreSQL or Microsoft SQL Server database. You need the latest copy of this file anytime you make configuration changes, like these:
- Database credentials
- Certificate updates
- Changes to authorization policy
If you plan to encrypt database connections, set up your PostgreSQL or SQL Server deployment for TLS.

The Hybrid Data Security setup process creates an ISO file. You then use the ISO to configure your Hybrid Data Security host.

podman rmi ciscocitg/hds-setup:stable  
podman login docker.io -u hdscustomersro
dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo
podman pull ciscocitg/hds-setup:stable
podman run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable

What to do next

Follow the remaining steps in Create a Configuration ISO for the HDS Hosts or Change the Node Configuration to create or change ISO configuration.

Use OpenSSL to Generate a PKCS12 File

Before you begin

OpenSSL is one tool that can be used to make the PKCS12 file in the proper format for loading in the HDS Setup Tool. There are other ways to do this, and we do not support or promote one way over another.
If you do choose to use OpenSSL, we are providing this procedure as a guideline to help you create a file that meets the X.509 certificate requirements in X.509 Certificate Requirements. Understand those requirements before you continue.
Install OpenSSL in a supported environment. See https://www.openssl.org for the software and documentation.
Create a private key.
Start this procedure when you receive the server certificate from your Certificate Authority (CA).

1	When you receive the server certificate from your CA, save it as `hdsnode.pem`.
2	Display the certificate as text, and verify the details. `openssl x509 -text -noout -in hdsnode.pem`
3	Use a text editor to create a certificate bundle file called `hdsnode-bundle.pem`. The bundle file must include the server certificate, any intermediate CA certificates, and the root CA certificates, in the format below: `-----BEGIN CERTIFICATE----- ### Server certificate. ### -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ### Intermediate CA certificate. ### -----END CERTIFICATE----- -----BEGIN CERTIFICATE----- ### Root CA certificate. ### -----END CERTIFICATE-----`
4	Create the .p12 file with the friendly name `kms-private-key`. `openssl pkcs12 -export -inkey hdsnode.key -in hdsnode-bundle.pem -name kms-private-key -caname kms-private-key -out hdsnode.p12`
5	Check the server certificate details. `openssl pkcs12 -in hdsnode.p12` Enter a password at the prompt to encrypt the private key so that it is listed in the output. Then, verify that the private key and the first certificate include the lines `friendlyName: kms-private-key`. Example: bash$ openssl pkcs12 -in hdsnode.p12 Enter Import Password: MAC verified OK Bag Attributes friendlyName: kms-private-key localKeyID: 54 69 6D 65 20 31 34 39 30 37 33 32 35 30 39 33 31 34 Key Attributes: <No Attributes> Enter PEM pass phrase: Verifying - Enter PEM pass phrase: -----BEGIN ENCRYPTED PRIVATE KEY----- <redacted> -----END ENCRYPTED PRIVATE KEY----- Bag Attributes friendlyName: kms-private-key localKeyID: 54 69 6D 65 20 31 34 39 30 37 33 32 35 30 39 33 31 34 subject=/CN=hds1.org6.portun.us issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 -----BEGIN CERTIFICATE----- <redacted> -----END CERTIFICATE----- Bag Attributes friendlyName: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US subject=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 issuer=/O=Digital Signature Trust Co./CN=DST Root CA X3 -----BEGIN CERTIFICATE----- <redacted> -----END CERTIFICATE-----

What to do next

Return to Complete the Prerequisites for Hybrid Data Security. You will use the hdsnode.p12 file, and the password you've set for it, in Create a Configuration ISO for the HDS Hosts.

You can reuse these files to request a new certificate when the original certificate expires.

Traffic between the HDS Nodes and the Cloud

Outbound Metrics Collection Traffic

The Hybrid Data Security nodes send certain metrics to the Webex cloud. These include system metrics for heap max, heap used, CPU load, and thread count; metrics on synchronous and asynchronous threads; metrics on alerts involving a threshold of encryption connections, latency, or a request queue length; metrics on the datastore; and encryption connection metrics. The nodes send encrypted key material over an out-of-band (separate from the request) channel.

Inbound Traffic

The Hybrid Data Security nodes receive the following types of inbound traffic from the Webex cloud:

Encryption requests from clients, which are routed by the encryption service
Upgrades to the node software

Configure Squid Proxies for Hybrid Data Security

Websocket Cannot Connect Through Squid Proxy

Squid proxies that inspect HTTPS traffic can interfere with the establishment of websocket (wss:) connections that Hybrid Data Security requires. These sections give guidance on how to configure various versions of Squid to ignore wss: traffic for proper operation of the services.

Squid 4 and 5

Add the on_unsupported_protocol directive to squid.conf:

on_unsupported_protocol tunnel all

Squid 3.5.27

We successfully tested Hybrid Data Security with the following rules added to squid.conf. These rules are subject to change as we develop features and update the Webex cloud.

acl wssMercuryConnection ssl::server_name_regex mercury-connection

ssl_bump splice wssMercuryConnection

acl step1 at_step SslBump1
acl step2 at_step SslBump2
acl step3 at_step SslBump3
ssl_bump peek step1 all
ssl_bump stare step2 all
ssl_bump bump step3 all

Deactivate Multi-Tenant Hybrid Data Security

Multi-Tenant HDS Deactivation Task Flow

Follow these steps to completely deactivate Multi-Tenant HDS.

Before you begin

This task should only be performed by a Partner full administrator.

1	Remove all customers from all your clusters, as mentioned in Remove tenant organizations.
2	Revoke the CMKs of all customers, as mentioned in Revoke CMKs of tenants removed from HDS..
3	Remove all nodes from all your clusters, as mentioned in Remove a Node.
4	Delete all your clusters from Partner Hub using one of the following two methods. Click on the cluster you want to delete, and select Delete this Cluster on the top right corner of the overview page. In the Resources page, click … on the right side of a cluster and select Remove Cluster.
5	Click on the Settings tab on the Hybrid Data Security overview page and click Deactivate HDS on the HDS Status card.

1	At your machine's command line, enter the appropriate command for your environment: In regular environments: `podman rmi ciscocitg/hds-setup:stable` In FedRAMP environments: `podman rmi ciscocitg/hds-setup-fedramp:stable` This step cleans up previous HDS setup tool images. If there are no previous images, it returns an error which you can ignore.
2	To sign in to the Docker image registry, enter the following: `podman login docker.io -u hdscustomersro`
3	At the password prompt, enter this hash: `dckr_pat_aDP6V4KkrvpBwaQf6m6ROkvKUIo`
4	Download the latest stable image for your environment: In regular environments: `podman pull ciscocitg/hds-setup:stable` In FedRAMP environments: `podman pull ciscocitg/hds-setup-fedramp:stable`
5	When the pull completes, enter the appropriate command for your environment: In regular environments without a proxy: `podman run -p 8080:8080 --rm -it ciscocitg/hds-setup:stable` In regular environments with an HTTP proxy: `podman run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In regular environments with an HTTPS proxy: `podman run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup:stable` In FedRAMP environments without a proxy: `podman run -p 8080:8080 --rm -it ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTP proxy: `podman run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTP_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` In FedRAMP environments with an HTTPS proxy: `podman run -p 8080:8080 --rm -it -e GLOBAL_AGENT_HTTPS_PROXY=http://SERVER_IP:PORT ciscocitg/hds-setup-fedramp:stable` When the container is running, you see "Express server listening on port 8080."

Thanks for your feedback.

Deployment Guide for Multi-Tenant Hybrid Data Security (HDS) (Beta)

New and changed information

New and changed information

Get started with Multi-Tenant Hybrid Data Security

Multi-Tenant Hybrid Data Security Overview

How Multi-Tenant Hybrid Data Security provides data sovereignty and data control

Limitations of Multi-Tenant Hybrid Data Security

Roles in Multi-Tenant Hybrid Data Security

Security Realm Architecture

Collaborating with Other Organizations

Expectations for Deploying Hybrid Data Security

High-level Setup process

Hybrid Data Security Deployment Model

Standby Data Center for Disaster Recovery

Proxy Support

Example of Hybrid Data Security Nodes and Proxy

Blocked External DNS Resolution Mode (Explicit Proxy Configurations)

Prepare your environment

Requirements for Multi-Tenant Hybrid Data Security

Cisco Webex License Requirements

Docker Desktop Requirements

X.509 Certificate Requirements

Virtual Host Requirements

Database server requirements

Additional requirements for Windows authentication against Microsoft SQL Server

External connectivity requirements

Proxy Server Requirements

Complete the Prerequisites for Hybrid Data Security

Set up a Hybrid Data Security cluster

Hybrid Data Security Deployment Task Flow

Perform initial set up and download installation files

Create a Configuration ISO for the HDS Hosts

Install the HDS Host OVA

Set up the Hybrid Data Security VM

Upload and Mount the HDS Configuration ISO

Configure the HDS Node for Proxy Integration

Register the first node in the cluster

Create and register more nodes

Manage Tenant organizations on Multi-Tenant Hybrid Data Security

Activate Multi-Tenant HDS on Partner Hub

Add tenant organizations in Partner Hub

Create Customer Main Keys (CMKs) using the HDS Setup tool

Remove tenant organizations

Revoke CMKs of tenants removed from HDS.

Test your Hybrid Data Security deployment

Test Your Hybrid Data Security Deployment

Monitor Hybrid Data Security Health

Manage your HDS deployment

Manage HDS Deployment

Set Cluster Upgrade Schedule

Change the Node Configuration

Turn off Blocked External DNS Resolution Mode

Remove a Node

Disaster Recovery using Standby Data Center

(Optional) Unmount ISO After HDS Configuration

Troubleshoot Hybrid Data Security

View Alerts and Troubleshoot

Alerts

Troubleshoot Hybrid Data Security

Other notes

Known Issues for Hybrid Data Security

Run HDS Setup tool using Podman Desktop

Use OpenSSL to Generate a PKCS12 File

Traffic between the HDS Nodes and the Cloud

Outbound Metrics Collection Traffic

Inbound Traffic

Configure Squid Proxies for Hybrid Data Security

Websocket Cannot Connect Through Squid Proxy

Deactivate Multi-Tenant Hybrid Data Security

Multi-Tenant HDS Deactivation Task Flow

Small Business

Enterprise

Devices

Solutions For

Resources

Company