Here’s a twist on network performance tests of the big three U.S. wireless operators: Fastest doesn’t always mean best.
That’s not exactly news, but it’s noteworthy when the winner of a network performance study happens to show up with the slowest network.
Signals Research Group (SRG) conducted an independent benchmark study of the AT&T, T-Mobile and Verizon wireless networks in the Dallas metro area in early December and gave AT&T the highest marks for overall network performance. Verizon came in second and T-Mobile was third in the overall performance category.
“Fun fact – the operator with the ‘slowest network’ came out on top in our study while the operator with the greatest 5G NR network coverage finished last,” SRG wrote in its February 17 report. “Furthermore, in many cases the operator with the most 5G NR network coverage delivered a poorer user experience when the smartphone used 5G NR or a mix of LTE and 5G NR in a given test than when the smartphone merely relied on LTE.”
In the category of voice performance, results were pretty close among the Big 3, with AT&T achieving 378 points; Verizon with 377; and T-Mobile with 375. In the category of data performance, the differences were bigger, with AT&T tallying 437 points, while Verizon had 388 and T-Mobile scored 365. The overall SRG ranking includes both voice and data performance.
“Having the fastest network is still nice to have and it’s great for marketing, but at the end of the day, it doesn’t necessarily define the user experience,” said Mike Thelander, CEO and founder of SRG.
RELATED: AT&T, T-Mobile, Verizon each win ‘fastest’ 5G awards: Huh?
While several firms measure the metrics of carriers’ networks, including speed and 5G availability, some use crowd sourcing apps and others, like RootMetrics, drive the nation’s highways. SRG decided to base its study on ETSI TR 103 559 recommendations.
“The purpose of the study was not so much to say, ‘this is the operator with the best network.’ It was to say, ‘these are the test criteria methodologies recommended by ETSI for benchmarking an operator’s network, and when you do that, these are the results,” Thelander told Fierce. “That was the main purpose of the study.”
ETSI stands for European Telecommunications Standards Institute, so obviously, it’s European focused, but it’s one of the driving forces behind the worldwide 3GPP and the standards community, and its influence extends well beyond Europe, he noted.
Although Covid contributed to a delay in publishing the results, “I don’t think the results would change directionally” if they were conducted today, according to Thelander. Operators might have added 5G coverage to the Dallas area in the past two months, but it’s not as though the presence of 5G always helps. In some cases, it can actually bring down speeds if introduced in lower spectrum bands.
RELATED: Verizon’s DSS performance ‘disappointing,’ says SRG
SRG’s summary of the study says it involved 115 hours of testing, 5,071 kilometers of driving, 1,300+ voice calls per network and 60,000+ data-related tests. The testing took place between December 2 and December 18 and used three Samsung Galaxy S20+ smartphones on each network.
They picked Dallas as the site mainly out of convenience; engineers from test and measurement company Rohde & Schwarz (R&S) were available there to collect the data on behalf of SRG, which was working remotely due to Covid restrictions.
Dallas just happens to be the location of AT&T’s corporate headquarters, so it’s probably happy to score high in its hometown. Fierce reached out to T-Mobile and Verizon for comment and will update this story with responses.
Testing, testing
By using the R&S Smart Analytics analysis tool, SRG examined the IP layer, RF metrics and the performance of the smartphones at the physical layer. As part of their analysis, they looked at LTE and 5G NR network coverage, including Band n71 and Band n41 (T-Mobile), as well as millimeter wave (mmWave) for Verizon.
According to the study, the Galaxy S20+ on the Verizon network used Band 48 (CBRS) more frequently than the Galaxy S20+ used Band 41 (2.5 GHz) on the T-Mobile network. “This outcome wasn’t due to poorer LTE Band 41 coverage, which was reasonably good, but to the network seemingly not reporting the Band 41 spectrum was available. We also do not believe Band 41 LTE is supported when Band n71 5G NR is active,” SRG wrote.
Benchmark campaigns based on crowd sourced data are fully independent and largely outside of the operator’s control, the report noted. However, operators do take steps with some test measurement applications to ensure they obtain the best possible results, even if the resultant performance metrics do not reflect a typical user experience when accessing content from the internet, according to the report.