Republic Wireless 11/29/18 outage update


#1

On Thursday, November 29, 2018 Republic Wireless experienced an outage impacting calling, messaging, and voicemail.

During the outage, some members asked what caused the outage. It’s a good question, and we’d like to provide some insight about what we experienced.

Republic Wireless partners with industry leaders to efficiently make use of existing infrastructures and resources for many of our services. An obvious example is that we don’t build our own cell towers across America; we contract with two major carriers to provide cellular service. Another easy-to-understand example is this Member Community. We didn’t build our own forum software from scratch; we contract with an excellent forum software company and they provide the platform and support to keep our Member Community running. The service outage yesterday occurred when a data center we partner with experienced a disruption so unusually severe that it impacted even our backup and failover systems. As the data center recovered, there were ripple effects across our systems that our engineers worked on throughout the evening and monitored closely into the night.

While our engineers continue their work to ensure stability across our service, there is also much that will be done to study yesterday’s events and look for ways to avoid or minimize the risk of recurrence.


Update: 12/3/2018
As we are seeing some concern from our members about the idea of a disruption at a data center, our leadership team has asked me to clarify that there was no data breach during this outage.


Outage affecting calling, texting, voicemail, Anywhere, activation (11/29/2018)
#2

Thank you for being so transparent with your customers as to the cause of yesterdays kerfuffle! As much as we don’t want them to happen…outages are part of our reality. Having RW constantly notifying it’s customers on the status of the situation gave me a sense of assurance that the issue was being handled.

Thank you, RW! Thank you!


#3

5 posts were split to a new topic: Two carrier partners, who?


#6

Thanks for the note to restart phone to connect with the post-outage-fix world. Joy now.

Doug


#8

So, all your backup system are in the same data center as the main one? That’s not really a true backup.


#9

I’m trying to understand why the complaints and negativity from some of you. This is just merely an update from the company as to what happened during the outage yesterday.


#10

I just got the Email from Doc Shufelt of R.W. this afternoon.
I appreciate the heads-up on this issue, and all you guys do to keep us up and running. Hopefully this is a “learning” experience and you will build in some redundancy for future occurrences.
It would be nice to give us a day of credit on next month’s bill.
Thanks R.W.!


#11

Is there a way to get notifications when issues arise with republic? I think that would be a nice feature, being able to know when Republic is experiencing issues. I know there is the status page, but something pushed would be beneficial.


#12

The main issue with something pushed is the issues may prevent the push out right [I would expect that to have been the case with this lady outage calling, texting, Republic anywhere connection, activation where all effected and I would expect a push notification or email would have also been effected.]
My recommendation would be to bookmark the status page/

I will also note that there will always be a small delay from Republic acknowledging an issue from when it 1st occurres as 1st reports will be treated as one off events until there are any reports and then scale and scope of an issue before Republic can send out a status, cfrom what I saw the 1st reports of this last event where around 2pm and Republic’s 1st status update was around 2:45


#13

I agree with what you are saying and I have the status page bookmarked, but I would have to check the status page all the time for issues.

I’m not saying immediately notify me, but some kind of notification that there is problem would be nice. Maybe having a republic status twitter feed specific to status updates (not the generic republic wireless twitter) that I could subscribe to or some other republic owned forum page that could be subscribed to.


#14

but you did miss the point about relaying on a service to push a notice that a said service is down is unreliable at best
when the service is down the ability of using that service to inform the end customer that the service is down doesn’t work


#15

yea. u cant send a email to someone notifying them that the email is down and not working…because of that exact reason.


#16

I understand that. I’m suggesting other ways of notifications as well. I’m not saying text me if text messages don’t work. I get that.

They do update the status page at some point I assume. And posting something on Twitter has nothing to do with republics ability for service.


#17

this was posted on social media both Facebook and twitter at 2:58 pm during the event


#18

So your technical response to people is to tell us that your partner had (a disruption so unusually severe that it impacted even our backup and failover systems.) I myself find this a non technical reason for an outage that took down even your backup system, bad PR on your part… So what caused the failure - An inbound gateway router or a security router or a BGP router or a power failure in the what seems like a single point of failure to me where their backup power system did not come online to keep all the routers - switches and ATM equipment going. IMHO and probably only IMHO you probably don’t really have true redundant links - router, servers etc.


#19

It seems to me that Republic has provided far more explanation that most carriers do when there’s an outage. I can’t imagine, nor have I ever seen, a cellular provider (rather than an ISP or data center provider) provide the type of data that you’re asking about.


#20

A post was split to a new topic: Still having issues - can’t call, text, no voicemail


#23

Hi @rickstr,

Thanks for your feedback and insight. I’m sorry to read that the information didn’t satisfy your thirst for knowledge. It wasn’t meant to be a technical response, but, as I indicated “some insight.” The specifics around the disruption would be details the partner would need to release, as they are not details that are ours to go into.

As for redundancy and failover, I would not mention them if they didn’t exist, but, as I pointed out initially, we do see opportunities for improvement and will be using this experience as a foundation for building that improvement.

It sounds like you have a lot of experience in networking and systems engineering. I hope you’ll share some of that expertise in our Member Community, as we often have networking questions as new members seek advice to best optimize their Wi-Fi networks for their Republic Wireless service.


#24

Hi @southpaw,

Thanks for the quick response. I myself being Cisco, Juniper, 3-com, Nortel, FORE Systems ATM and more for 15+ years and was a manager of a networking team / department. However, now retired for 10 years (most say i am retarded, haha) So yes i do have exp in networking / systems engineering and design. Anyway you did mention (As for redundancy and failover, I would not mention them if they didn’t exist) well, ) Well, if true redundancy was in effect at that partner then no interruption of service to your customers (not me) would have taken place. Only imo. Yes their are them freak catastrophic failures (like backup power not working when utility power goes down) or fail-over configuration from one router to another not working or not configured correctly so when one link goes down it does not switch or ospf failure. Yes i know how partners don’t want to give technical details about failures. Kind of makes them look bad. Have a good day.


#25

This is just not correct. Having spent the better part of my career in data center services I’ve seen, more than once, a bad routing table update cascade across multiple data centers and take down primary, secondary and tertiary backups.