Webex for BroadWorks Troubleshooting Processes

Escalating an Issue

After you have followed some of the troubleshooting guidance, you should have a reasonable idea of where the issue is rooted.

1

Collect as much information as you can from the systems related to the issue

2

Contact the appropriate team at Cisco to open a case (see Contacts section)

What Client Information to Collect

If you think you need to open a case or escalate an issue, collect the following information while troubleshooting with the user:

  • User identifier: CI email address or User UUID (this is the Webex identifier, but if you also get the user's BroadWorks identifier, that will help)

  • Organization identifier

  • Approximate time frame during which the issue was experienced

  • Client platform and version

  • Send or collect logs from the client

  • Record the tracking ID if shown on client

Check User Details in Help Desk

1

Sign in to https://admin.webex.com/helpdesk.

2

Search for and then click the user. This opens the user summary screen.

3

Click the user name to see the detailed user configuration.

Useful information in this view includes the user’s UUID, common identity (CI) cluster, Webex app cluster, Calling Behaviour, BroadWorks account GUID.

4

Click Copy if you need to use this information in another tool, or attach it to a Cisco case.

View Customer Organization in Help Desk

1

Sign in to https://admin.webex.com/helpdesk.

2

Search for and then click the customer organization name.

3

Scroll down until you see Customer Portal View and click View CustomerName to see a read-only view of the Customer org – including users and configuration.

Retrieve User Logs from Partner Hub

When troubleshooting desktop and mobile client issues it is important for Partners (and TAC) to be able to view the client logs.

1

Ask the user to Send Logs.

2

Ask the user to Export the Calling Environment send you the ced.dat file.

3

Get the client logs from Partner Hub or Help Desk (see below).

Partner Hub option:

  1. Sign in to Partner Hub and find the user’s Customer Organization.

  2. Select Troubleshooting.

  3. Select Logs.

  4. Search for the user (by email).

  5. View and download the client logs as a zip file.

Help Desk option:

  1. Sign in to Help Desk.

  2. Search for the organization.

  3. Click the organization (opens up the summary screen).

  4. Scroll down to click View customer.

  5. Select Troubleshooting.

  6. Select Logs.

  7. Search for the user (by email).

  8. View and download the client logs as a zip file.

Client Check for Calling Service

1

Sign in to the Webex client.

2

Check that the Calling Options icon (a handset with a gear above it) is present on the sidebar.

If the icon is not present, the user may not yet be enabled for the calling service in Control Hub.

3

Open the Settings/Preferences menu and go to the Phone Services section. You should see the status SSO Session You're signed in.

(If a different phone service, such as Webex Calling, is shown, the user is not using Webex for BroadWorks.)

This verification means:

  • The client has successfully traversed the required Webex microservices.
  • The user has successfully authenticated.
  • The client has been issued a long-lived JSON web token by your BroadWorks system.
  • The client has retrieved its device profile and has registered to BroadWorks.

Get Client Logs or Feedback

  • See the Resources section to find specific client logs on Webex desktop clients, or ask users to send logs.

  • Ask users of mobile clients to send logs, then you can get them via partner hub or help desk.


Send logs is silent. However, if a user sends feedback, it goes to the Cisco Webex app devops team. Be sure to record the user’s feedback number if you want to follow up with Cisco. For example:

Get Calling Environment Data

Webex client logs are heavily redacted to remove personally identifiable information. You should export the Calling Environment Data from the client in the same session as you notice the issue.

1

In the client, click the profile picture, then click Help > Export Calling Environment Data.

2

Save the resulting file ced.dat for troubleshooting calling issues for this user.

Important: Logout from or restart the client clears the internal cache. If you export ced.dat after that, the exported data will not correspond with any logs that were sent before the cache.

Reset Webex Database

1

On the client click Help > Health Checker.

2

Select Reset Database.

This triggers a full reset of the client and loads the Webex app login screen.

Verify that Webex Should Register to BroadWorks

The Webex app checks the following information to determine whether to register to BroadWorks:

  • User entitlement to broadworks-connector

  • Calling behavior for organization and user

Check a user’s calling behavior and connector entitlement

  1. Sign in to Help Desk (https://admin.webex.com/helpdesk) with your partner administrator credentials.

  2. Search for the user.

  3. Click on the user and check the Calling Behavior entry. It should be "Calling in Webex".

  4. Click the user name to open the User Details screen.

  5. Scroll down to locate the entitlements section, and verify that broadworks-connector is included.


    A Webex for BroadWorks user should NOT have the bc-sp-standard entitlement if they are intending to use Webex for BroadWorks. This is the entitlement for “Webex Calling (Broadcloud)” which is Webex app calling through a Cisco-managed cloud calling service.

Check the organization’s calling behavior

  1. Sign in to Help Desk (https://admin.webex.com/helpdesk) with your partner administrator credentials.

  2. Search for the organization.

  3. Click on the organization and check the Calling Behavior entry. It should be “Calling in Webex”.

Analyze PSLog for User Provisioning Issues

Use the Application Server’s PSLog to see the HTTP POST request to the provisioning bridge and the response from Webex.

In a correct working case, the response is 200 OK and after a few minutes you can see the user - and new Customer org if it is first user – has been created in Webex.

You can verify this by searching Help Desk for the email address you see in the POST.

Before you begin

Collect a PSLog from the Application Server during a flowthrough provisioning attempt with a test user.

1

The first thing to check is the HTTP response code:

  • Anything other than 200 OK is a user provisioning failure.

  • 200 OK could still indicate a failure if something about the subscriber profile does not work in the Webex services upstream of the provisioning bridge.

  • 400 may contain a message node in the response. The provisioning bridge could not process something in the subscriberProfile. There may be something wrong with the subscriber details, or incompatibility with a setting in the template.

  • 401 means the provisioning credentials entered on the AS do not match those entered on the template in Partner Hub.

  • 403 could indicate something misconfigured on Application Server. Check the target of the request. it should not be an IP address, it should be the provisioning bridge URL you can see on your template in Partner Hub.

  • 409 indicates a conflict between the supplied subscriberProfile and existing Webex data. There may be an existing user with that email address. Check the message in the response.

2

You can also check the original HTTP POST for any suspect values that could cause provisioning to fail.

The POST contains a subscriberProfile XML structure. Inside this, useful nodes to check are:

  • bwuserid: Use this to find the subscriber profile if you need to edit it in BroadWorks.

  • group: If the template is in "Service Provider mode", this is lowercased and becomes the name of the Customer org you see in Partner Hub.

  • serviceProvider: If the template is in "Enterprise mode", this is lowercased and becomes the name of the Customer org you see in Partner Hub.

  • primaryPhoneNumber: Must exist. Provisioning fails without it.

  • email: Becomes the user ID in Webex. Must be valid and unique to Webex, otherwise provisioning fails.


 

Ignore the services stanza: it is created by AS, and accepted but not used by Webex.

Analyze XSP Logs to Troubleshoot Subscriber Log in

This flow describes BroadWorks Authentication mode. You can see the authentication mode on the BroadWorks Template, in Partner Hub. See Configure your Customer Templates in https://help.webex.com/en-us/z9gt5j/Webex-for-BroadWorks-Solution-Guide#id_137726.

The following ladder diagram shows the interaction between the user, client, Webex services, and BroadWorks system, when the user is doing BroadWorks authentication in the Webex app. Also, the connection between Webex and the XSP is secured by MTLS.

The discussion that follows explains what you can expect to see when investigating the logs for a successful login.

Figure 1. BroadWorks Authentication and Device Configuration

User interacts with client, client interacts with Webex services:

  • The user supplies their email address to the Webex app (1 in diagram).

  • CI knows to redirect this user to enter their BroadWorks password (via UAP) (2 in diagram).

  • The IDP Proxy submits a get profile request to the Xsi interface on the XSP.

In the tomcat access_log:

  • Look for the GET request for the subscriber profile, from Webex towards the Xsi-Actions interface (2.1 in diagram). It has the Webex user ID. E.g.

    GET /com.broadsoft.xsi-actions/v2.0/user/webexuserid@example.com/profile

In the XsiActionsLog:

  • Look for the profile GET request from Webex (2.1 in diagram). It has the Webex user ID. E.g.

    GET /com.broadsoft.xsi-actions/v2.0/user/webexuserid@example.com/profile

    The headers include authorization: Basic and user-agent: broadworksTeamsClient

  • The XSP then does OCI-P Basic authentication against BroadWorks (AuthenticationVerifyRequest and AuthenticationVerifyResponse, like any other application doing basic authentication via Xsi) and also a UserGetRequest and ServiceProviderGetRequest to gather the subscriber information.

  • The Xsi response to Webex contains an XML Profile block containing the (BroadWorks) userId and other details (2.2 in diagram).

Client and Webex services interactions:

  • IDP proxy matches user profile received from BroadWorks and issues SAML assertion to client (2.3 in diagram)

  • Client exchanges SAML assertion for a CI token (3 in diagram)

  • The client checks that the signed in user has the broadworks-connector entitlement (4 in diagram). You can check user entitlements in Help Desk)

  • Client uses CI token to request a JSON Web Token (JWT) from IDP proxy (5 in diagram)

  • IDP proxy validates CI token at CI

  • IDP proxy requests JWT from authentication service

In the authenticationService log:

  • Look for the token request from Webex (5.2 in the diagram), e.g.:

    GET /authService/token

    which has http_bw_userid header and others.

  • The XSP does OCI-P UserGetLoginInfoRequest, to validate that the supplied user id corresponds to a BroadWorks user (5.3 in diagram). AuthService has established trust with Webex by virtue of the mTLS connection, so can issue LLT.

  • Look for the response (5.4 in diagram) from LongLivedTokenManager - Token generated, subject: bwksUserId@example.com, issuer: BroadWorks …

    and StatusCode=200 which you can associate with the original request using the trackingid: CLIENT… header.

In the XsiActionsLog:

  • The client is now able to present the long-lived token at Xsi-Actions interface to get its device profile (6 in diagram). E.g.:

    GET /com.broadsoft.xsi-actions/v2.0/user/bwksUserId%40example.com/profile/device

    With the headers authorization: Bearer token and user-agent: WebexTeams (variant/version)

  • The Xsi-Actions interface POSTs the token to the authservice (configured to be on the loopback interface) e.g.: 127.0.0.1:80 POST http://127.0.0.1:80/authService/token

    which you can correlate with the trackingid: CLIENT… header in the GET and the X-BROADSOFT-CORRELATION-ID : CLIENT… header in the POST.

In the authenticationService log:

  • The receipt of the POST from Xsi (loopback)

  • A StatusCode=200 back to Xsi

  • And a token validation response, having a "token" JSON block in the body.

  • Correlated using the trackingid: CLIENT…

In the XsiActionsLog:

  • Having received 200 OK from authservice, which validated the client’s token, the Xsi-Actions application now sends OCI-P request for UserPrimaryAndSCADeviceGetListRequest

  • Receives OCI-P UserPrimaryAndSCADeviceGetListResponse containing the accessDeviceTable XML structure.

  • The OCI-P response is encoded as Xsi response to client, including AccessDevices XML structure, which has the deviceTypes e.g. Business Communicator – PC and the urls where the client can retrieve the device configuration files.

Client continues as normal:

  • Selects a device entry and interacts with DMS to get device profile (6 in diagram)

  • Registers to BroadWorks via SBC retrieved in configuration from DMS (7 in diagram)