Prepare Your Environment

The Broadworks XSPs can be deployed directly on the Internet, or can reside in a DMZ fronted by a load balancing element such as the F5 BIG-IP. To provide geo-redundancy, the XSPs can be deployed in two (or more) datacenters, each can be fronted by a load balancer, each having a public IP address. If the XSPs are behind a load balancer, the Webex microservices and App see only the IP address of the load balancer and Broadworks appears to have just one XSP, even if there are multiple XSPs behind.

In the example below, the XSPs are deployed at two sites, Site A and Site B. There are two XSPs fronted by a Load Balancer at each site. Site A has XSP1 and XSP2 fronted by LB1, and Site B has XSP3 and XSP4 fronted by LB2. Only the Load Balancers are exposed on the public network, and the XSPs are in the DMZ private networks.

DNS Configuration

The Webex Cloud microservices must be able to find the Broadworks XSP server(s) for connecting to the Xsi interfaces, authentication service and CTI.

Webex Cloud microservices will perform DNS A/AAAA lookup of the configured XSP hostname and connect to the returned IP Address. This could be a load balancing edge element, or it could be the XSP server itself. If multiple IP Addresses are returned, the first IP in the list will be selected. SRV lookup are not currently supported.

Example: The partner’s DNS A Record for discovery of Round-Robin balanced internet-facing XSP server/Load Balancers.

Failover

When the Webex microservices send a request to the XSP/Load Balancer and the request fails, several things can happen:

• If the failure is due to a network error (ex: TCP, SSL), the Webex microservices mark the IP as blocked and immediately perform a route advance to the next IP.

• If an error code (HTTP 5xx) is returned, the Webex microservices mark the IP as blocked and immediately perform a route advance to the next IP.

• If no HTTP response is received within 2 seconds, the request times out and the Webex microservices mark the IP as blocked and perform a route advance to the next IP.

Each request is tried 3 times before a failure is reported back to the microservice.

When an IP is in the blocked list, it will not be included in the list of addresses to try when sending a request to a XSP. After a predetermined period of time, a blocked IP expires and goes back in the list to try when another request is made.

If all IP addresses are blocked, the microservice will still try to send the request by randomly selecting an IP address from the blocked list. If successful, that IP address is removed from the blocked list.

Status

The status of the connectivity of the Webex Cloud services to the XSPs or Load Balancers can be seen in Control Hub. Under a BroadWorks Calling Cluster, a connection status is displayed for each of these interfaces:

• XSI Actions

• XSI Events

• Authentication Service

The connection status is updated when the page is loaded or during input updates. The connections statuses can be:

• Green: When the interface can be reached on one of the IPs in the A record lookup.

• Red: When all IPs in the A record lookup are unreachable and the interface is unavailable.

The following services use the microservices to connect to the XSPs and are impacted by the XSP interface availability:

• Webex App Login

• Webex App token refresh

• Untrusted email/self activation

• Broadworks Service Health Check

DNS Configuration

The Webex App accesses the Xtended Services Interface (XSI-Actions & XSI-Events) and Device Management Service (DMS) services on the XSP.

To find the XSI service, the Webex App performs DNS SRV lookup for _xsi-client._tcp.<webex app xsi domain>. The SRV points to the configured URL for the XSP hosts or load balancers for the XSI service. If SRV lookup is not available, the Webex App falls back to A/AAAA lookup.

The SRV can resolve to multiple A/AAAA targets. However, each A/AAAA record must map to a single IP address only. If there are multiple XSPs in a DMZ behind the load balancer/edge device, it is required that the load balancer be configured to maintain session persistence to route all requests of the same session to the same XSP. We mandate this configuration because the client's XSI-event heartbeats must go to the same XSP that is used to establish the event channel.

In Example 1, the A/AAAA record for webex-app-xsp.example.com does not exist, and does not need to. If your DNS requires that one A/AAAA record must be defined, then only 1 IP address should be returned. Regardless, the SRV must still be defined for the Webex App. If the Webex App uses the A/AAAA name that resolves to more than one IP address, or if the load balancer/edge element does not maintain session persistence, the client eventually sends heartbeats to an XSP where it did not establish an event channel. This results in the channel being torn down, and also in significantly more internal traffic which impairs your XSP cluster performance. Because the Webex Cloud and Webex App have different requirements in A/AAAA record lookup, you must use a separate FQDN for the Webex Cloud and Webex App to access your XSPs. As shown in the examples, Webex Cloud uses A record webex-cloud-xsp.example.com, and Webex App uses SRV _xsi-client._tcp.webex-app-xsp.example.com.

Example 1—Multiple XSPs, each behind separate load balancers

In this example, the SRV points to mutiple A records with each A record pointing to a different load balancer at a different site. The Webex App will always use the first IP address in the list and will only move to the next record if the first is down.

Example 2—Multiple XSPs behind a single load balancer (with TLS Bridge)

For the initial request, the load balancer selects a random XSP. That XSP returns a cookie that the Webex App includes in future requests. For future requests, the load balancer uses the cookie to route the connection to the correct XSP, ensuring that the event channel doesn't break.

DMS URL

During the login process, the Webex App will also retrieve the DMS URL to download its config file. The host in the URL will parsed and the Webex App will perform DNS A/AAAA lookup of the host to connect to the XSP that hosts the DMS service.

Example: DNS A Record for discovery of Round-Robin balanced internet-facing XSP server/Load Balancers by Webex App to download config files through DMS:

How Webex App Finds XSP Addresses

The client attempts to locate the XSP nodes using the following DNS flow:

1. Client initially retrieves Xsi-Actions/Xsi-Events URLs from Webex Cloud (you entered them when creating the associated BroadWorks Calling Cluster). The Xsi hostname/domain is parsed from the URL and the client performs SRV lookup as follows:

   1. Client performs an SRV lookup for _xsi-client._tcp.<xsi domain>

   2. If the SRV lookup returns one or more A/AAAA targets:

      1. The client does A/AAAA lookup for those targets and caches the returned IP addresses.

      2. The client connects to one of the targets (and therefore its A/AAAA record with a single IP address) based on the SRV priority, then weight (or at random if they’re all equal).

   3. If the SRV lookup does not return any targets:

      The client does A/AAAA lookup of the Xsi root parameter and then tries to connect to the returned IP address. This could be a load balancing edge element, or it could be the XSP server itself.

      As noted, the A/AAAA record must resolve to one IP address for the same reasons.

2. (Optional) You may subsequently provide custom XSI-Actions/XSI-Events details in the device configuration for the Webex app, using the following tags:

   <protocols>
   	<xsi>
   		<paths>
   			<root>%XSI_ROOT_WXT%</root>
   			<actions>%XSI_ACTIONS_PATH_WXT%</actions>
   			<events>%XSI_EVENTS_PATH_WXT%</events>
   		</paths>
   	</xsi>
   </protocols>

   1. These configuration parameters take precedence over any configuration in your BroadWorks Cluster in Control Hub.

   2. If they exist, the client will compare against the original XSI address it received via the BroadWorks Cluster configuration.

   3. If there is any difference detected, the client will re-initialize its XSI Actions/ XSI Events connectivity. The first step in this is to perform the same DNS lookup process listed under step 1 – this time requesting a lookup for the value in the %XSI_ROOT_WXT% parameter from its configuration file.

      Make sure to create the corresponding SRV records if you use this tag to change the Xsi interfaces.

Failover

During login, the Webex App performs a DNS SRV lookup for _xsi-client._tcp.<xsi domain>, builds a list of hosts, and connects to one of the hosts based on the SRV priority, then weight. This connected host becomes the selected one for all future requests. An event channel is then opened to the selected host and a heartbeat is sent regularly to verify the channel. All requests sent after the first one include a cookie that is returned in the HTTP response, therefore, it’s important that the load balancer keeps session persistence (affinity) and always sends requests to the same backend XSP server.

If a request or a heartbeat request to a host fails, several things can happen:

• If the failure is due to network error (ex: TCP, SSL), the Webex App route advances immediately to the next host on the list.

• If an error code (HTTP 5xx) is returned, the Webex App marks that IP address as blocked and route advances to the next host on the list.

• If a response is not received within a period of time, then the request is considered failed due to timeout and the next requests are sent to the next host. However, the timed out request is considered as failed. Some requests are retried after failure (with increasing retry time). The requests that the assumed non vital are not retried.

When a new host is tried successfully, it becomes the new selected host if the host is present in the list. After the last host in the list is tried, the Webex App will rollover to the first one.

In the case of heartbeat, if there are two consecutive request failures, the Webex App will re-initialize the event channel.

Note that the Webex App does not perform fail-back, and DNS service discovery is performed only once at sign-in.

During sign-in, the Webex App tries to download the config file through the XSP/Dms interface. It performs a A/AAAA record lookup of the host in the retrieved DMS URL and connects to the first IP. It will first try to send the request to download the config file using a SSO token. If this fails for any reason, it will try again but with the device username and password.