Diagnosing a troubled SIP call has a tendency to be a real pain. Whether it’s running wireshark, tcpdump, or collecting debugs, having to sort through duplicate packets and attempting to merge different pcap files together does not provide a simple way to troubleshoot a single call while looking at both sides of the call in a single ladder diagram.

Fortunately the Acme Packet SBC now includes a free tool embedded in the code that once enabled, allows it turns the SBC into a SIP capture device. The distinct advantage here is seeing both sides of a call in a single ladder diagram. Even better, extra (and very useful) information is included in between each step of the ladder diagram referencing internal “logic decisions” as they occur as traffic passes through the SBC.  Finding a particular capture is easy using Search Filters which allow you to specify just about any criteria.

Pop-up context provides tool tips and additional information about a call depending on what area you are hovering over. It is also possible to export a capture locally so it may be emailed and viewed by others  rather than having a variety of users logging into the SBC’s web interface. Alternatively, captures may be exported as ASCII text files with proper and readable formatting of the call information.

There are three main parts to viewing a captured call. The first is the Session Summary view which contains information such as source and destination IP addresses, URI’s, Realms, etc..

 

The second viewing pane is SIP Message Details. This is the actual ladder diagram and SBC events.

 

The third pane is for viewing QoS statistics such as jitter, packet loss, delay, and MOS scores for the specified call.

In order to enable web browser viewing of SIP Monitoring and Tracing the web server must be set to the enabled state.

ATL# configure terminal
ATL(configure)# system
ATL(system)# web-server-config
ATL(web-server-config)# state enabled

The next step is to create one or more filters. In the following example there is a filter called hostedIpPbx and in the user portion of the filter any SIP messages containing the phone number digits 781801 will be captured.

The next step is to enable sip-monitoring and identify which monitoring filter should be used. Applying the filter here enabled the filter globally on the system. Usually filters are best applied to specific realms or session agents (under monitoring-filters) to capture only interesting traffic.

In this particular network there are IP phones behind a Cisco ASA firewall which NAT to the public internet and need to register to a SIP softswitch which is being “hidden” behind the SBC in a Service Provider core network.

IP_Phone——Cisco_ASA——-[APKT_SBC_Public<>APKT_SBC_Private]—-SIP_Registrar
<–Customer_Prem->NAT>           <–SIP Monitoring and Tracing–>
192.168.1.198    72.12.135.253    72.12.135.250      10.12.135.250       10.12.135.140

Click on the image below to see a full screenshot from the SBC’s SIP Monitoring and Tracing tool showing a SIP phone behind the ASA registering to a SIP Registrar server. As you can see SM&T captures both the ingress and the egress side and displays into one simple ladder diagram. The Session Summary provides additional useful information specific to the SBC that would normally not be seen in 3rd party capture tools.

 

  1. Help says:

    I am trying to turn on the web server and under system it is not available. Do i have an older version? or do i need a license?

    COR(system)# web-server-config
    % command not found

  2. Mark says:

    SIP Monitoring and Tracing was released in version 6.3.9 which is available from their customer portal. It is supported on the 3800, 4500, and the (Linux based) Server Edition and VMWare editions!

  3. Help says:

    Thanks looks like we’ll need a upgrade of code.
    ACMECOR# sh ver
    ACME Net-Net 3800 Firmware SCX6.1.0 MR-4 Patch 2 (Build 682)
    Build Date=03/31/10

  4. Jerry says:

    Hi Mark,

    I am looking for a Voice SP solutions, i am hear about the broadworks, is that a great softswitch for Service Provider side? This is in addition with APKT 4500.

    Kind Regards

  5. Milan says:

    Hi,
    We are looking for the monitoring and reporting solution for our CUBE-SP.
    We need to monitor call statistics of our different adjacencies i.e total calls, successful calls, failed calls and call details.

    Do we have any solution for that?

  6. Mark says:

    The only platform I am aware of is from Secure Logix, but it is intended for fraud prevention by anlyzing call processing through CUBE. It’s not intended for troubleshooting call flows.

  7. Dirk says:

    Hi Mark,

    First I must say great site, it’s already been a helper for me 😉

    My question towards you is about version NNSCX6.3.9 P7, whats your experience with the 6.3 releases? Are they stable enough?

    I’m very interested in the SIP trace/monitoring feature and I would like to implement this at our customer. They have NN3820’s with the current version NNSCX620M11P1, is it wise to step up to version 6.3.9 P7?

    Thanks,

    Dirk

  8. Mark says:

    Hi Dirk. 6.3.9p7 is very stable. Keep in mind there were different versions of 6.3.x before reaching 6.3.9, so it’s well baked.

  9. Dirk says:

    Thanks for the quick reply 😉

    I’ve asked Acme Packet a few questions also, but did not receive a conclusive answer.
    So the field experience from others is well appreciated.

    Do you know what the base version was for version 6.3.9P7?
    We found an issue in 620M11P1 (tsipd crash due to null pointer exception Ticket 45111).
    Normally they will implement all fixes into new releases, however in the 6.3.9/6.3.x release there is no reference to previous issues.

    So that’s why we are in doubt for the release 6.3.9 P7. The feature set is very nice, but if we could encounter issues it’s not an option.

    Kind Regards,

    Dirk

  10. Jim Barber says:

    Hi Mark,

    Thank you so very much for putting this site up. It is a great resource and I am sure that many people appreciate it even if they do not voice it.

    I am trying to implement this for all sip calls. What would we use for a wild card that would represent all users? I have tried this:

    filter-config
    name All
    address 0.0.0.0
    user *
    sbc(filter-config)#

    And it does not seem to work. I enabled the filter but to no avail.

    sbc(sip-monitoring)# show
    sip-monitoring
    state enabled
    monitoring-filters All
    trigger-window 0
    sbc(sip-monitoring)#

    Any ideas?

  11. Mark says:

    Hi Jim,

    SIP Monitoring & Tracing isn’t specifically intended to capture all SIP traffic all the time. As the network grows, it will begin to consume more resources if the traffic gets heavy. That being said, in my lab, I created a monitoring-filter that I apply to the Core (private) realm of the SBC facing my SIP server. For example, if my SIP server is 10.12.135.140/24, I create a filter for 10.12.135.0/24. It doesn’t matter what Access realm the traffic comes in on, all traffic the has a next-hop to the Core realm and destined for network 10.12.135.0/24, the SBC will capture the SIP traffic and it’s viewable in SM&T. You can create multiple monitoring filters under filter-config, then go to the realm-config, and use the + symbol to add multiple filters to a single realm.

  12. Jim Barber says:

    Thanks,

    That is really cool. I will definitely look into creating the filters the way you suggest. As for why my captures were not working….

    I found the reason. Feel pretty sheepish now. I forgot to execute the actual capture.

    like so:

    sbc# capture start global *

    this command actually fires off the capture. Working fine now.

  13. Hello Mark,
    I just found your site on a search. You provide some great information and solved a problem for me here. Thank for this. I’ll come back more often and recommend you to associates now.
    Thank,
    Victor

  14. Adam Geffner says:

    Hi Mark,

    The problem we have is that when you go to the ACME via browser session and look at the session ladder stats, it only holds about 30 min worth of calls. Where do I configure to extend the time duration or cache size of calls it can hold? Alternatively I suppose would be to be able to export this data to a NAS drive, then be able to painlessly retrieve it.

    P.S. I was going to ask Ron, but I think he has his hands full lately.

    Thanks – Adam

  15. Mark says:

    Depending on how much traffic you have on the SBC and if you are capturing ALL traffic versus INTERESTING traffic using the filter, the SBC could potentially fill up quite fast. Make sure you only have the filter applied on one realm to reduce the number of captures and avoid duplicates. If you have a 4500 there is an option to add an internal hard drive kits for expanded storage (originally designed for higher CDR capacity).

  16. Denis says:

    Hello.
    Thank you very much for the great blog. Actual answers and a lot of useful info!

    Next version of 4500 OS (6.4p2) have no web server (((.

    Do you know, why they take it out?…

  17. Mark says:

    Hi Dennis,

    Right now you are seeing SCX6.4 posted on the Acme Packet portal which SCX is the traditional stream of code. There is a target date of April 24th to release another stream of 6.4 using the ECX prefix. ECX6.4 series will contain the Web GUI. In addition to SIP Monitoring and Tracing, you will have the ability to save/delete/restore backup configs, access log files, configure the SBC in an Expert mode similar to the command line structure but through the Web GUI, upload LRT files, and more.. Stay tuned!

    The web interface and the provided tools do use some additional CPU overhead/cycles. When SIP Monitoring and Tracing was introduced it was designed for very specific call capturing (replacing packet-trace) but many customers tend to use it for capturing all the traffic on the network. Acme Packet Palladion (formerly IPTEGO) is better suited for this. This one of the primary reasons why SCX has removed the web services.

  18. Mike P says:

    Thanks for this information. We upgraded our 4500 last night. Do you know how far back the history is allowed to go on the call details? Trying to find a setting for this and have been unsuccessful.

  19. Young says:

    I am on version: ACME Net-Net 4500 Firmware SCX6.2.0 MR-6 Patch 1 (Build 855)
    Build Date=05/05/11

    The only thing I have is: notify sipd siplogs
    Is there away to do filtering when enabling siplogs to filter a specific user?
    Currently sipmsg.log max file size grows to 1Mb and goes to sipmsg.log.1 -> .9
    Can I even change this, so that I can capture wider time range?
    thx
    Young

  20. roger says:

    i am very fond of your website.sir how do we capture pcap traces in a sbc?

  21. Alino says:

    Hello mark,
    i followed all these steps, but which command should i use to “display” the filtered traffic?
    in order words, after setting up all required filtered, how do i display data in a ladder diagram from the command line?
    thanks

  22. Mark says:

    You don’t display SM&T captures from the command line. You need to enable the web-server-config element as shown. You then use your web browser to log into the SBC’s GUI to use the SM&T tools for viewing call captures.

  23. Mark says:

    The packet-trace command will send traffic you’ve identified as interesting to Wireshark through the management interface. There is an older post on the blog on how to do this. However, this is being replaced by SIP Monitoring and Tracing which is a 100% onboard tool for capturing and viewing traffic through the SBC’s web interface.

  24. NP says:

    Hi Mark,

    Thank you very much for the great blog.
    I am currently having SBC 9200 on SD7.1.0 MR-1 Patch2 (Build 469) – 09/20/11 – Current config version: 1163.

    I am unable to configure the WEB server on above version.
    Do you know if it’s going to be available for 9200 ? and version?

    Thanks,
    NP

  25. Mark says:

    Thank you for checking out the blog. Unfortunately the 9200 does not support this feature.

  26. Alino says:

    Hello Mark,
    Have you already set-up session replication within SBC? I have the license in my lab here, and the config seems straightforward,
    I put a PC as the CRS ( Call Recording Server),
    but when i go on that PC, and launch wireshark, I was expecting to be able to see captured SIP packets that are going back and forth thru the SBC? i’m not seeing anything.
    question: what do i need to do on the CRS to see such capture traffic?
    thanks

  27. Sandeep says:

    Hi Mark,
    My SBC is currently on 6.4 version, but I don’t see this capability under system menu. Is this a licensed feature?

    ACME Net-Net 4500 Firmware SCX6.4.0 MR-3 Patch 2 (Build 341)
    Build Date=11/21/13

    Thanks

  28. Mark says:

    SCX640 is the only version where Acme Packet did not implement it after introducing it in 6.3.9. It’s in the following releases. SCX6.3.9 and newer 6.3.x releases, ECX6.4, SCZ7.1.x (Currently 6300, 6100, and 4500 support SCZ7.1.2, 3800 and 1100 support is coming soon).

  29. JuanJ says:

    Hey Mark, great and usefull post… Just have a question, I just set all the steps at the SBC. but… how do I look at the web? whats the address.. or how do I setup in order to viiew on web page or something…

    regrds

  30. Mark says:

    The commands to setup SIP Monitor and Trace are in the first tutorial that was posted. Note the web-server-config command. http://www.markholloway.com/blog/?p=1974

    The web interface IP is associated with the management interface of the SBC. The login credentials are the same as the SSH login. If you haven’t changed the default then it should be admin/packet

  31. Ann O'Nymous says:

    Nice tool, I guess Acme put it in in response to their competitors, notably the Service Assurance stuff on Perimeta (Metaswitch)? IMHO the latter still knocks the socks off the embedded webserver. No need to wait for a repro, and processing happens off-box. https://www.youtube.com/watch?v=kNjj-xGKlxU

  32. Iniyan says:

    How to enable the trace permanently?After reboot the SBC I have to start again manually the trace in the system.Any command even after reboot the trace will work.

    The commnd am using enable trace – capture start global *

  33. Gaven says:

    I have the following ACME SBC -Acme Packet Net-Net 3820 ECX6.4.0 MR-3 Patch 2 (Build 391) and I would like to find out the following:
    1. How do I verify the maximum simultaneous sessions at a given time

    2. How many times per day/week/month we are reach or get close to the current maximum possible sessions.

    3. Is this information easily extracted from the SBC`s?

  34. Donald says:

    Hi Mark,

    I tried enabling this on our 3820 SCz7.2.0, as instructed, but I’m not seeing any output. I’ve place User=* in the filter. Should I be more specific on the user? Is this the reason why?

    Thanks,

  35. Adrian says:

    Hi Mark,

    Do you know how to erase all previous captured traces in the sbc web interface?

    Thank you so much, your site is awesome.

  36. Mark says:

    Are you talking about SIP Monitor and Trace? They are buffered and there is no way to manually delete. A reboot is the only way. I believe the SBC’s retain 4000 records max on 7.x software before cycling through.

  37. Mark says:

    To clarify, you’ve created the filter and then applied it to a realm and you are still not seeing anything?

  38. Mark says:

    The 3820 supports up to 8,000 concurrent calls. Show SIP Sessions should give you the output you are looking for. The SBC supports SNMP polling so you could trend over time. It also generate CDR’s and HDR’s

  39. Juan says:

    Just found this bit of information regarding SIP Monitoring on version S-CZ7.3.x.

    http://docs.oracle.com/cd/E67973_01/doc/sbc_scz730_releasenotes.pdf

    SIP Monitor and Trace / WebGUI
    The SIP Monitor & Trace and WebGUI features are unsupported. Ensure that the system > web-server-config > state parameter is set to disabled.

  40. Bob Holtzman says:

    I am trying to run the SM&T tool but can’t locate the path. Would you know the reason why the web-server-config path is not under ‘system’? We are running the following software:

    ACME Net-Net 3820 Firmware ECX6.4.1 MR-1 GA (Build 14)

  41. Mark says:

    That looks like the FIPS version. I don’t think it’s on that release.

  42. Bob Holtzman says:

    Is there a way to remove the Admin Security license on the 4500? I think that may be what is restricting access to the web-server-config. At least temporarily?

  43. Mark says:

    Unfortunately no. As part of the FIPS Level 1 certification it canot be removed.