Overview

Troubleshooting for Webex Calling allows you to troubleshoot media quality and call connectivity-related issues for specific Webex Calling calls. You might want to troubleshoot calls for various reasons, such as complaints from customers on how they're experiencing poor call quality. In this case, you can go directly to the troubleshooting view, search for those specific calls, and then view the hop details to pinpoint where the issue could be. You can also use troubleshooting to proactively monitor call quality for the organization and get ahead of specific problems before users notice.

To access the troubleshooting view, you must have the full admin, read-only admin, or support admin role in Control Hub.

You can search using the following criteria to get a list of calls where a media session was used with at least one Webex Calling-registered endpoint:

  • Email IDs

  • Phone numbers

  • MAC address

  • Call IDs

  • Correlation ID

Partial search is supported for email IDs, phone numbers, and call IDs. For phone numbers, you can enter the first three to eight digits and then press Enter to see matching entries. If you enter more than eight digits, the search will attempt to find an exact match. Note that if you copy and paste a phone number from elsewhere, you must include the +1 for the search to work.

Webex Calling troubleshooting allows you to:

  • Search for successful, failed, and refused calls.

  • View the end-to-end experience of the participants of the call.

  • View a hop detail of the call.

  • View if the media traverses through the Webex Calling cloud, or directly between the users (using Interactive Connectivity Establishment (ICE)).

  • View Insights, if there’s no media in the call, or when the path optimization setup was unsuccessful.

  • View calls for the past 21 days.

  • View signaling-related call failures and rejections.

  • Analyze the call quality metrics that impacted the experience of the user. For example, an administrator may observe high jitter on clients over Wi-Fi networks, but packet loss and latency may be acceptable.

  • Detect if the issue is with the caller or the callee.

The calls using Webex Calling appear after the call ends.

The troubleshooting view helps to identify the problem area by providing all the relevant metrics and can’t necessarily provide you the root cause for a poor call. Look at these pointers to identify various factors and determine the resolution options:

  • The end-to-end experience of the user.

  • The Hop Details view.

  • Send or receive metrics from the user or the media relay point.

  • Whether the call occurred to or from an external network or between the Webex Calling registered endpoints.

Search view

Searched calls in Troubleshooting in Control Hub

When you search for a call, the following data is available:

Column nameDefinition

Call status

If the call attempt was successful or not. When the call attempt has a Refusal or Failure status, you can hover over the status to see the relevant code. Possible values are:

  • Successful—Call attempt was successful.

  • Refusal—Call attempt was refused.

  • Failure—Call attempt failed.

Quality

The Webex Calling calls are graded for quality. However, for the Webex meetings or Call on Webex sessions, this grading doesn’t apply. The call experience is graded as:

  • Poor—indicates that either the caller or the callee had a poor end-to-end experience (for example, choppy audio).

  • Good—Indicates the end-to-end experience for the caller and the callee didn’t exceed thresholds.

  • None / Not available—No media quality data available.

Service

Service used for the call.

Start Time

Start time of the call, based on the time zone selected next to the date range selector.

Meeting / Calling number

Telephone number of the calling party.

Name

Display name of the caller and callee.

Host / Caller

Display name of the caller.

Participants

Number of users in the call.

Duration

Duration for how long the call lasted.

Site / Location

Webex Calling location of the caller.

Conference / Correlation ID

Correlation ID to tie together multiple call legs of the same call session.

Searched user's Key Performance Indicators (KPIs)

KPIs for a searched user in Troubleshooting in Control Hub

When you search for a user, the following KPIs are available:

  • Failed calls—Number of calls that failed over the selected date range.
  • Poor meeting minutes—Number of minutes during Webex Meetings that had poor quality performance over the selected date range.
  • Call on Webex with poor quality—Number of Call on Webex calls that had poor quality performance over the selected date range.
  • Webex Calling calls with poor quality—Number of Webex Calling calls that had poor quality performance over the selected date range.
  • Poor WebRTC minutes—Number of minutes that had poor quality while using WebRTC over the selected date range.
  • Total meetings minutes—Total number of Webex Meetings minutes over the selected date range.
  • Total Call on Webex call minutes—Total number of Call on Webex minutes over the selected date range.
  • Total Webex Calling call minutes—Total number of Webex Calling minutes over the selected date range.
  • Total WebRTC minutes—Total number of WebRTC minutes over the selected date range.

Hop details view

Example of a Webex Calling call's hot details in Troubleshooting

The following data is available in the hop details view:

TermDefinition
Endpoint

Displays one of the following:

  • Desk Phone for a physical endpoint

  • Webex App

  • Room Device for Cisco Room Series devices

Hardware

Displays one of the following:

  • Desk phone model information for a physical endpoint

  • “-“ for a Webex App

  • Room Series model information for Cisco Room Series device

Location

Webex Calling location that's configured for the user. The country that Webex Calling was provisioned in is also shown in brackets.

Local IP

The local IP address of the client for the network interface that it is using to transmit media. IP addresses are partially masked to preserve the personal identity of users.

Public IP

This is the public IP address of the client as seen by the cloud. For enterprises this is the address of the firewall providing the NAT. IP addresses are partially masked to preserve the personal identity of users.

MAC addresses

The MAC address of the client endpoint.

Geolocation

Geo lookup of Public IP address. This address is not accurate, if connected over PNC. If the user is using the Webex App and connecting to the enterprise through a VPN, the location is not accurate.

ISP

Internet Service Provider who provides network connectivity to the Webex Calling Cloud.

Network

The type of network connection that the client used to exchange media. Possible values are:

  • Wi-Fi

  • Ethernet

  • Cellular

  • Unknown

Audio Codec

(Send or Receive) The media encoding and decoding format in use for the media that are transmitted by a client.

Video Codec

(Send or Receive) The media encoding and decoding format in use for the media that are transmitted by a client. Applies only for a video call.

SIP Session ID

Original (or initial) and Final Session IDs.

Trunk

Only available when a local gateway is involved with the hop. Name of the inbound and outbound trunk.

Route Group

Only available when a local gateway is involved with the hop. Name of the route group.

PSTN Vendor Name

Only available when PSTN is involved with the hop. Name of the vendor.

Some metrics are masked in the article screenshots to preserve the identity of the user.

From the hop details, you can:

  • View if the hop was routed through PSTN or a local gateway.

  • View the type of user that made or received the call. Some examples include HuntGroup, VoiceMailGroup, and RoutePoint. The Detailed Call History report provides a list of all the possible user types.

  • View insights about the call in these scenarios:

    • There was no-media for any of the hops related to the call.

    • The Path optimization setup was unsuccessful.

    • One of the hops failed or refused the call. Failed hops are marked as red and refused hops are marked as yellow. A reason is also given on why the hop failed or was refused.

  • Hover on the device to view the end-to-end call experience.

  • Hover on the hop between the endpoint and Webex Calling cloud to view the hop details.

The end-to-end call experience is based on the media quality data that is collected from each Webex Calling registered endpoint (Webex App or device such as 8865 or Desk Pro) at the end of the call. The call is graded as good if the endpoint has the following thresholds:

  • Packet loss less than 5%.

  • Latency or Round Trip Time (RTT) less than 400 ms.

  • Jitter less than 150 ms.

The quality of the hop is derived from data that are collected from the media relay point in the Webex Calling cloud. For PSTN calls through CCPP or local gateway, the data collection is from the Webex Calling cloud and not from the PSTN endpoint. A hop is graded as good if the media relay point has the following thresholds:

  • Packet loss less than 2.5%.

  • Latency or RTT less than 200 ms.

  • Jitter less than 75 ms.

You can also save the data in the hop details by selecting Actions and then choosing one of the following options:

  • Save troubleshooting call records (CSV)
  • Save call report (JSON)
  • Save participant details (CSV)

Call Details pane

The following data is available in the Call Details right pane of the hop details view:

TermDefinition
Correlation IDCorrelation ID to tie together multiple call legs of the same call session.
Calling DateThe date when the call occurred.
Calling TimeThe time of when the call started and ended, shown in the time zone that you selected in the search view.
Session Type

The type of session that is supported. For example: Webex Call

ParticipantsThe number of participants who joined the call.
Caller NameName of the caller.
Caller EmailEmail address of the caller.
Caller NumberPhone number that the caller used during the call.
AudioThe type of audio used.
VideoDisplays Yes if video is enabled by a participant. If video wasn't enabled at all, it shows No.
Path Optimization

Specifies if the call path optimization applies to the call. The accepted values are:

  • ICE (Interactive Connectivity Establish)

  • PNC (Private Network Connect)

  • No optimization

Calling Type

Calling Type can be one of the following:

  • Emergency

  • Enterprise

  • International

  • Mobile

  • National

  • Operator

  • Premium

  • Shortcode

  • Tollfree

  • Unknown

  • URI

Call Ended by

User who ended the call.

Dialed Digits

The keypad digits as dialed by the user, before pre-translations.

Related reason

Indicates a reason for why the hop was created. For example, a value of CallQueue indicates that the hope is due to a call attempt from a call queue to an agent. You can find more information about related reason values in the Webex Calling detailed call history report.

Access the Webex Calling troubleshooting view

Partial search is supported for email IDs, phone numbers, and call IDs. For phone numbers, you can enter the first three to eight digits and then press Enter to see matching entries. If you enter more than eight digits, the search will attempt to find an exact match. Note that if you copy and paste a phone number from elsewhere, you must include the +1 for the search to work.

To analyze a Webex call, perform the following:

1

Sign in to Control Hub and go to Monitoring > Troubleshooting.

2

Search for the email ID, phone number, MAC address, call IDs, or the correlation ID you’d like to view.

You can group related calls together in the search view by using the correlation ID.

3

Apply a filter of your choosing. The filters available are:

  • Location
  • Personal quality
  • Device type
  • Service type

Filters aren't available if you search for phone numbers or correlation IDs.

4

Click a specific call to inspect the hop details view.

Export Troubleshooting call details as a CSV file

You can export Troubleshooting call details as a CSV file. Exporting these call details is useful for complex calls involving multiple hops, as it allows you to identity issues or patterns at a glance. You can export up to 100 Troubleshooting call details per CSV file.

1

Sign in to Control Hub and go to Monitoring > Troubleshooting.

2

Once you have a list of calls that you want to view, select Export > Save troubleshooting call records (CSV).

Highlighted Export button for searched calls in Troubleshooting in Control Hub

Troubleshooting media quality issues

The hop-by-hop view helps you to locate where the problem occurred. Now that you’ve found where the problem is, and with the metrics (jitter, packet loss, latency) you can try the following to troubleshoot the issue.

Typical possibilities for media issues are:

  • Network/ISP/location specific issues—Due to firewall, network configuration or bandwidth there’s a pattern of poor experiences in a particular location or network subnet. Use the per call troubleshooting view (identify the location associated with the poor session) with the analytics view to review the aggregate patterns for the location.

  • User specific issues—A user or device is connected on a poor network (for example, Wi-Fi or working from home) which means their experience is impacted by the associated network capabilities. See the Use CScan to Test Webex Calling Network Quality article to identify the issue with the network.

  • Call type specific issues—A user’s poor experience is because of the quality on the far end. This is typical in PSTN scenarios where the user is talking to another user on a mobile network and the session has high packet loss on the PSTN network.

  • No-media issue—There may be no media transmission in some hops. The Insights banner displays the cause at the top of the Hop details page and the liable party in the information box of the hop detail page. Some of the possible causes for no-media in calls along with liable parties are listed here:

    • Webex not receiving media from the sender.

    • Webex not receiving media from the receiver.

    • Webex not receiving media from either direction.

    • Webex not sending media to the receiver. Webex engineering addresses this issue.

    • Webex isn’t receiving media from Cloud PSTN. Webex engineering addresses this issue.

    • Webex isn’t receiving media from cloud service. Webex engineering addresses this issue.

    • Webex isn’t receiving media from Local gateway. Customer administrator must investigate the issue.

  • Media Path Optimization Failure—Few calls cannot successfully set up media path optimization. The Insights banner displays the cause for unsuccessful ICE calls and the resolution at the top of the Hop details page.

    Some of the possible reasons are:

    • ICE unsuccessful due to stun server access - see Webex Calling Port Reference Information.

    • ICE unsuccessful due to connectivity check - verify connectivity between networks.

    • ICE unsuccessful as the default path round trip time was similar/better than any optimized path.

Limitations

Media quality metrics and originating and terminating flows from the network side aren’t available for the following devices:

  • Analog phones

  • Third-party devices

  • IPv6 endpoints

Supported Call Flows

The media quality reports are collected from the caller and callee endpoints and the media relay points. This allows a segmentation of the media experience to narrow down and identify whether the issue occurred at the:

  • Caller or callee

  • Media path to or from the Webex Calling cloud.

Call legs appear if there was a media session that is established with at least one Webex Calling registered endpoint on the call. For example, for an outbound call from hunt group to eight agents, if only one agent answer, then there is no media experience to troubleshoot for the other seven agents.

There are five types of media experiences or paths for Webex Calling troubleshooting, which are:

  • On-net Optimized – Calls within the organization where ICE is successful and the media flows directly between the users. See Webex Calling Media Optimization with Interactive Connectivity Establishment (ICE) for detailed information.

  • On-net Unoptimized – Calls within the organization where Interactive Connectivity Establishment (ICE) was not possible or established. In this scenario, the media flows through Webex calling cloud.

  • On-net Cloud Hosted – Calls within an organization where media is provided by a media server that is hosted in the cloud (for example, listening to voicemail, dialing into an auto attendant).

  • Off-net Call to or from the Webex Calling registered endpoint -

    • via Cloud Connected PSTN Provider- Inbound and outbound calls of an organization where the other party is on the PSTN network. The media is relayed through a cloud connected PSTN provider (CCPP), over a high-quality interconnect.

    • via Local Gateway- Inbound and outbound calls of an organization where the other party’s media is through the enterprise. Behind the local gateway the media session can be from enterprise hosted user (for example, registered to call control in the enterprise) or from PSTN, where PSTN is provided by the enterprise.

If there are 1 or 2 Webex Calling registered users who are involved in a point-to-point on-net call, then the troubleshooting view presents metrics for one or both the parties. If the call is off net (User 1 receives a call from a PSTN user), then the troubleshooting view presents only User1's client metrics, along with the metrics taken from the media relay point.

Most of the call scenarios in the troubleshooting view show two call legs (caller and callee); however, some of the call scenarios (such as call park or retrieve) show only one call leg. In such cases, the other call leg shows up separately in the troubleshooting view. This does not hinder troubleshooting the call and detecting where the problem occurred. However, it does require the admin to manually correlate the two call legs using a common entity such as overlapping time. Future enhancements to the troubleshooting view will eliminate the need to use manual lookups.

The hop metrics vary during a session depending on the sampling time and variability in the network. The values that are reported by the media relay point and the clients (end-to-end experience) may not align. However, they should be in close alignment to allow segmentation along the path.

We recommended using the individual call troubleshooting view with the aggregate information that is derived from Analytics.

Let’s analyze the call quality for the different call types using the troubleshooting view.

On-net Optimized Call Flow

Calls within the organization where ICE is successful and the media relay in the cloud is removed from the path. The media flow is directly between the user's devices.

Inference:

1

The call is graded as good as both the caller and callee have good end-to-end experience.

2

The administrator can observe that media flows directly between the two users and does not travel through Webex calling cloud.

Optimized call flows can have a poor experience, if the enterprise or local network is the source of the issue, since the media between the two users will traverse the local network. Latency or RTT is always lower on an optimized call but packet loss and jitter can still be a factor depending on the network between the two users.

On-net Unoptimized Call Flow
Calls within the organization where an ICE call isn’t successful and the media flows through the Webex Calling cloud.

Inference:

The administrator can observe the following:

1

The caller’s end-to-end experience is graded as poor.

2

There is a problem with the caller's network hop that affects both the send and the receive stream.

3

The callee’s network hop does not have an issue.

4

The callee’s end-to-end experience is graded as poor due to the issue from the caller.

On-net Cloud Hosted Call Flow
Calls within an organization where the media is provided by a media server that is hosted in the cloud.

Inference:

The call is graded as good as the caller’s end-to-end experience is within the accepted thresholds. The administrator can observe that the following:

1

The network hop for the caller is graded as poor as some of the metrics are above the acceptable threshold.

2

The send stream from the voicemail is graded as good per the metrics collected from the media relay point.

3

The media server used to collect or deposit the voicemail does not currently report metrics. However these servers are hosted and managed as part of the Webex Calling Cloud so the quality of that link segment is internal and always high quality, low latency.

4

The administrator can observe the caller’s end-to-end experience is graded as good although the hop is graded as poor. This is due to the callee's hop having a good network that compensates for the performance degradation of the caller's network.

Off-net PSTN through CCPP
Calls in and out of an organization where the other party is on the PSTN network. The media is relayed from a cloud connected PSTN provider (CCPP).

In the example, the client media is coming from a PSTN provider through the cloud.

The call is graded as poor as the callee’s end-to-end experience is not within the accepted thresholds. The administrator can observe the following:

1

The problem is with the caller’s PSTN hop specifically with the sending stream.

2

The callee’s network hop does not have an issue.

3

The callee’s end-to-end experience is graded as poor due to the issue from the caller.

4

The end-to-end experience and the receive stream metrics of the caller who is on the PSTN network are not currently available as these metrics are not transmitted to the Webex Calling Cloud.

Off-net PSTN through Local Gateway

Calls in and out of an organization where the caller's media is coming from an enterprise. The media session can be from an enterprise hosted user (for example, registered to UCM) or from PSTN, where the PSTN is provided through the enterprise.

Inference

The call is graded as poor as the caller’s end-to-end experience is not within the accepted thresholds. The administrator can observe the following:

1

There is a problem with the caller’s hop to the Webex Calling Cloud both on the send and receive stream.

2

The caller’s end-to-end experience is graded as poor either due to the issue observed in the hop or the issues at the user’s end (devices, network, and so on.)

3

The incoming traffic to the Webex Calling Cloud from the callee’s end is graded as good.

4

The end-to-end experience and the receive stream metrics of the callee who is called from the local gateway are not currently available as these metrics are not transmitted to the Webex Calling Cloud.

Troubleshooting failed and refusal calls

The Call Status column indicates if a call was successful, refused, or failed. When a call has a Refusal or Failure status, you can hover over the status to see the relevant code.

When you drill down to a user's list of calls, a Failed call KPI is available to quickly identify if that user is experiencing significant issues. You can hover over the Refusal or Failure status to view the specific code associated with each status.

In the hop details view, a banner is displayed to show you more information about why the call has a Refusal or Failure status. You can also see if the call status happened during the originating or terminating hop.

Example of failed call scenarios

Failure Code: No route to destination

Failure code: no route to destination error code example for a call in Troubleshooting

A call attempt was made by a workplace device but failed due to having no route available to the destination.

Failure Code: Protocol error

Failure code: protocol error code example for a call in Troubleshooting

A call attempt was made by a workplace device but failed due to a protocol-related error.

Failure Code: Internal error

Failure code: internal error code example for a call in Troubleshooting

A call was made from a local gateway but failed due to internal errors with the workplace device at the destination. The call was established for 14 minutes before resulting in an internal error.

Example of refusal call scenarios

Refusal Code: Call rejected

Hop detail view with a call rejected refusal code in Troubleshooting in Control Hub

A call attempt failed because the attempt couldn't reach the destination or the interface to the destination wasn't functioning correctly. Examples of rejected scenarios are the following:

  • Call attempt is rejected due to anonymous call rejection.

  • Call attempt is rejected due to incoming call permissions.

  • Call attempt is rejected due to outgoing call permissions.

  • Call attempt is rejected due to outgoing call permissions.

  • Call attempt is rejected due to call intercept configured.

  • Call attempt is rejected due to call park or call retrieve failure.

  • Call attempt is rejected due to push-to-talk rejection.

  • Call attempt is rejected due to selective call acceptance.

  • Call attempt is rejected due to selective call rejection.

  • Call attempt is rejected due to errors like unknown number, insufficient permissions, etc.

Failure Code: Unassigned number

Refusal code: unassigned number error code example for a call in Troubleshooting

A call attempt was made by a caller but was rejected because the destination was to an unassigned number.

Call outcome reasons

The following table lists the Refusal and Failure codes available for unsuccessful calls.

Call outcome reasonDefinition

Success

Call was successful. Possible codes are:

  • Normal—Call completed successfully.
  • UserBusy—Call is a success, but the user is busy.
  • NoAnswer—Call is a success, but the user didn't answer.

Refusal

Call was refused. Possible codes are:

  • CallRejected—Call attempt rejected at the recipient's end.
  • UnassignedNumber—The dialed number isn't assigned to any user or service.
  • SIP408—Request timed out because couldn’t find the user in time.
  • InternalRequestTimeout—Request timed out as the service couldn’t fulfill the request due to an unexpected condition.
  • Q850102ServerTimeout—Recovery on timer expiry or server timed out.
  • NoUserResponse—No response from any end-user device or client.
  • NoAnswerFromUser—No answer from the user.
  • SIP480—Callee or called party is currently unavailable.
  • SIP487—Request is terminated before the call was answered.
  • TemporarilyUnavailable—User is temporarily unavailable.
  • LocalGatewayLoop—Loop detected between the local gateway and Webex Calling.
  • AdminCallBlock—Call attempt is rejected due to the organization's call block list.
  • UserCallBlock—The call to user is rejected because the number is on the user's block list.
  • Unreachable—Unable to route the call to the desired destination.
  • UserAbsent—User is temporarily unreachable or unavailable.
  • Undefined—None of the above reasons.

Failure

Call failed. Possible codes are:

  • DestinationOutOfOrder—Service request failed as the destination can’t be reached or the interface to the destination isn’t functioning correctly.
  • SIP501—Invalid method and can’t identify the request method.
  • SIP503—Service is temporarily unavailable so can’t process the request.
  • ProtocolError—Unknown or unimplemented release code.
  • SIP606—Some aspect of the session description wasn't acceptable.
  • NoRouteToDestination—No route available to the destination.
  • Internal—Failed because of internal Webex Calling reasons.
  • MaxConcurrentTerminatingAlertingRequestsExceeded—Number of simultaneous unanswered calls to a local gateway, for the same calling and called number, exceeded the limit.
  • Undefined—None of the above reasons.