Comcast is boldly going into uncharted waters with its deployment of virtual cable modem termination systems (vCMTS), and it has learned a few lessons along the way.
During last week's SCTE Cable-Tec Expo, several Comcast executives on a panel provided updates on Comcast's roll outs of virtual cable modem termination systems, although Comcast's Jan Hofmeyr, executive vice president and chief network officer, declined to say how many vCMTS had been deployed to date.
"I will tell you that we are actively deploying the virtualized CMTS platform while the vast majority of our subscribers are still on the traditional CTMTS platforms," he said. "As for the challenges, I will start with a couple of things. We are literally writing the book as we go, okay? I can't pick up the phone, and call Cisco's John Chapman, and say 'Hey, what have you guys done?' because they haven't done it either. And so the challenge is how do we learn how to build and how do we learn how to operate at the same time?
"The second challenge is, as you're dealing with the smaller big forms, you're starting to go into race conditions that you don't see at the big appliance level, but when you start going into the individual containers you do. So it's a lot of remote and theoretical corner cases that you start to run into. It's a combination of these two that are the most critical in terms of the challenges we've had deploying."
The good news on the second challenge is that containers and applications that aren't functioning properly can be killed off and then new ones can take their places. Comcast Fellow Bob Gaydos, who was speaking on the panel that Hofmeyr was moderating, said that with cloud computing, containers and service providers can create and kill microservices as needed to scale.
Comcast is serving more than 26 million broadband customers with its legacy CMTS. Comcast Cable's Elad Nafshi, senior vice president, Next Generation Access Networks, said there are "very long innovation cycles that are ultimately driven by proprietary chipsets that are powering these (CMTS) devices." One of Comcast's goals for vCMTS is to increase the overall reliability of the network.
"In terms of service reliability, today's platforms are really based around polling," said Nafshi, who was also on the panel. "We poll our CMTSs every 15-20 minutes, and that's what gives us the picture of the health of the access network, but the reality is that there's a whole bunch of stuff that can happen in those 15-20 minutes that go completely unnoticed because we just have no way of telling it. "
In order to do maintenance, Comcast employees currently have to go through command-line interface (CLI) commands and then cut and paste the views every 16 seconds to see if the host devices are back online. By contrast, a hyperscale company such as Amazon Web Services, can get real time telemetry down to the second, Nafshi said.
"When you look at our competitors out there, Facebook, Google, et cetera, they innovate in web speed," he said. "They're reliant on Intel-based hardware that gets twice as powerful every 18 months, that's Moore's Law. So if you're not able to take advantage of that and have the software to go with it, then we're not able to keep up with the web speed competition.
"It (virtualization) opens up a world of resources that are not dedicated to Broadcom-based development. We're dealing with broad web-based technology, and at the end of the day it's about reliability, it's about speed of innovation, and its about scale, that really drives and interests us towards why we need to make the sweep into virtualization."
Gaydos said that Comcast thought about putting its vCMTS in Amazon Web Services' cloud, but shipping all of the data back and forth from AWS' cloud to Comcast would impact the time it takes for it to run in real time.
"So we run it in our own private cloud, yet all of our ELK (Elasticsearch, Logstash, and Kibana) stack, for example, is sitting in the public cloud," Gaydos said. "So we ship all our logs there (to the public cloud). Let them (cloud providers) deal with all of the hyperscale, the data storage, while we focus on the low latency, high bandwidth applications."
Gaydos said there would always be technical challenges when a new platform is being developed. By working with Intel, Comcast was able to resolve its software issues related to deploying virtualized CMTSs.
RELATED: Cisco executives: Disaggregation of tech is easy, business cases not so much
"Realistically, because Intel came in and helped us and really helped us make the vCMTS work on their platform, that's not the hard part," he said. "The hard part is actually the operations because when we go in to do the CMTS, it's not just about virtualizing the CMTS. We're actually disaggregating the CMTS and we're using DAA nodes, so now you have these digital nodes that are in the field that take time to boot and they don't come on instantaneously. So trying to convince the line techs that they can't just unpower a node and repower the node and the modes are there in 10 seconds, those are parts of our challenges.
"There's no CLI anymore because it's spread across containers so there's no place to log into. So getting our operators used to our dashboards that we're building in real time and looking at the streaming telemetry rather than getting their hands on and doing the commands that they're used to doing, it's a culture shock. This doesn't just affect the people building the platform, it affects the people running the platform more than anything else because how they did their job yesterday is different from how they're going to do their job today."
Gaydos said Comcast had training programs in place to reskill employees on how to work with the virtualized CMTSs, which in theory free those techs up to perform other functions for Comcast such troubleshooting services in customers' homes or finding where noise leakage comes into the network.