Teams Direct Routing with Ribbon 8k - How to reboot using CLI / WebGUI is not loading

Recently got an alert due to connectivity of the Ribbon 8k (strange), nothing wrong with networking. However, SolarWinds shows a failure.  

End up checking the SBC via GUI and got following 502 Bad gateway,

SBC is running but no idea why GUI failing, in this situation very limited things can do. Started with Putty and connected via console. 

Following are the command to find the status of the SBC VMs status 

Sudo -i 

Virsh list - Will help me to understand status of the SBCSWeEdge 

Podman ps - Will help me to understand status of the OS, dockers/container status

sys-tools debug - Will help me to run debug via CLI 

Anyway above cmds not helping to resolve the problems, but i end up running a reboot middle of a day :( 

systemctl reboot

This was help to bring back GUI after few minute time. While this is happening all the calls routed via DRaaS. We are using Call Tower as vendor still provide DRaaS even though NEW customer required to go with Operator Connect for Teams. 

Additional info

I have checked the debug logs (Ribbon 8K) couldnt found any storage issues.

Filesystem               1K-blocks    Used Available Use% Mounted on

devtmpfs                      4096       0      4096   0% /dev

tmpfs                      8172244    4944   8167300   1% /dev/shm

tmpfs                      3268900    9896   3259004   1% /run

efivarfs                       128      43        81  35% /sys/firmware/efi/efivars

/dev/mapper/vg00-root     36798464 6819696  29978768  19% /

/dev/sda4                   983040  279988    703052  29% /boot

/dev/sda3                   613160    7220    605940   2% /boot/efi

/dev/sda1                  8993268 6738272   1776568  80% /e8k-data

/dev/loop0                 3378444 3378444         0 100% /LocalRepos

/dev/mapper/vg00-storage   5054464   68300   4986164   2% /storage

/dev/mapper/vg00-var       2031616  851120   1180496  42% /var

/dev/mapper/vg00-var_log   2031616 1420064    611552  70% /var/log

/dev/mapper/vg00-var_tmp    983040   40116    942924   5% /var/tmp

/dev/mapper/vg00-tmp       2031616   47332   1984284   3% /tmp

‎Teams administrator permission doesn't allow to create/modify resource accounts anymore! - MC780743 (Updated) Changes to Administrator Permissions Required to Create and Manage Teams Resource Accounts

This started with MC780743 change and following will explain as a summary,

Summary

The message details an update on the rollout timeline for changes to Teams administrator permissions required for creating and managing Teams Phone Resource Accounts. The change, rolling out in December 2024, will require Microsoft 365 Global Admin or User Admin roles for these tasks, affecting Teams Administrator, Teams Communications Administrator, and Teams Telephony Administrator roles. Organizations should prepare by adjusting processes accordingly.

However this was pushed to some of my client tenant recently or many notice recently :). 

Issue

Before change people able to create Resource Accounts without having User Administrator Roles in Tenant and also able to modify those accounts. After MC780743 change, everyone required to have User Administrator Role and i had that attached to my admin account too. However, even after Teams administrator and User administrator both active i couldn't modify the resource accounts and It was giving following error. Basically i created the Resource accounts in bulk and assigned AA or CQ later.

Microsoft article - Manage resource accounts for service numbers - Microsoft Teams | Microsoft Learn

Solution

Surprise :D This was not documented by Microsoft article but randomly found old Skype for business related article during SfB and Teams beginning. Required to add Skype for Business administrator role in addition to "Teams administrator and User administrator"

• Teams administrator
• User administrator
• Skype for Business administrator
  
  

Found in old article Manage resource accounts for service numbers - Microsoft Teams | Microsoft Learn