June 25, 2017

Analysis: The year of the big stadium Wi-Fi upgrade

Carolina Panthers director of IT James Hammond shows off a new under-seat Wi-Fi AP at Bank of America Stadium. Credit: Carolina Panthers

Carolina Panthers director of IT James Hammond shows off a new under-seat Wi-Fi AP at Bank of America Stadium. Credit: Carolina Panthers

Even in the midst of several brand-new stadium debuts and the future-proofed wireless networks inside them, there is a separate, yet distinct trend emerging in the big-stadium, wireless connectivity world: Call it the year of the big upgrade.

Our profile in our latest STADIUM TECH REPORT of Bank of America Stadium in Charlotte, N.C., is a case in point: Thanks to the never-ending demand for more connectivity for fans, stadiums that deployed networks just a few years ago are now finding that those old systems already need upgrades or replacements, typically at a much higher cost than the original network. In addition to BofA Stadium, the New England Patriots’ home, Gillette Stadium, also got a Wi-Fi makeover this past summer, going from about 400 Wi-Fi APs to well over a thousand, with most of the new ones deployed under seats.

According to Fred Kirsch, who oversees the Gillette Stadium network, some of the under-seat placements there were especially tricky, since granite underneath the stands didn’t allow for the ability to drill through the concrete. A workaround involving an above-ground enclosure was envisioned and manufactured, underlining the custom complexity of network deployment found from stadium to stadium. No two are the same, and what works at one may or may not work at another.

But what is common across all these large venues is the ever-increasing need for bandwidth, a moving target that has yet to slow down or stabilize. Last year the story that turned everyone’s head was the need by carriers to upgrade their DAS infrastructure at Levi’s Stadium ahead of Super Bowl 50 – this coming just a year after the stadium had opened for business. While the demands of a Super Bowl (especially Super Bowl 50, which set records for DAS and Wi-Fi usage) are perhaps much different than everyday events, it’s still a safe bet that for many stadiums with Wi-Fi networks – especially the early movers – 2016 has become a year of reckoning, or biting the bullet and writing more checks for more coverage, perhaps seemingly too soon after the initial rollout.

Getting ready for Super Bowl LI

In Houston, NRG Stadium finally has Wi-Fi, and not a moment too soon, with Super Bowl LI on the near horizon. Since the venue didn’t have Wi-Fi prior to this season it’s not really an upgrade but it’s hard to understate the challenge of putting in a Super Bowl-ready network in just one summer, a construction calendar shortened by the fact that integrator 5 Bars and equipment vendor Extreme Networks had to wait until after the NCAA Men’s Final Four was over to begin installing cabling and APs. At of the start of the NFL season the Wi-Fi network is already live at NRG Stadium, and is sure to go through weekly tweaks as the league marches on toward its championship game.

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

Gillette Stadium before the Sept. 11 game vs. the Miami Dolphins. Credit: Steve Milne, AP, via Patriots.com

And while attention-grabbing new stadiums like US Bank Stadium in Minneapolis and Mercedes-Benz Stadium in Atlanta are planning big network capacity from the get-go, some new stadiums like T-Mobile Arena in Las Vegas have upgrade thinking planned in from the start, with the idea that the network will never really be a finished product, at least until they stop making new phones or developing new apps. Of course, that future isn’t happening anytime soon, with the Apple iPhone 7 announcement with the new double-lens camera coming in just before the start of another football season.

New phones and new apps mean more bandwidth demands, leading even those who already have stadium networks to keep wondering if what they’ve installed is enough. We suspect this may be an ongoing story line for the foreseeable future, so – stay tuned here to Mobile Sports Report for the latest success stories and lessons learned from those who have already jumped in or jumped back in to the deployment fray.

Editor’s note: This column is from our latest STADIUM TECH REPORT, which is available for free download from our site. Read about Wi-Fi deployments at Bank of America Stadium, Mercedes-Benz Stadium and more!

Niners fans use 2.76 TB of Wi-Fi for season opener at Levi’s Stadium

Niners fans celebrate a touchdown in the season opener. Credit: LevisStadium.com.

Niners fans celebrate a touchdown in the season opener. Credit: LevisStadium.com.

Fans at the San Francisco 49ers’ home opener used 2.76 terabytes of data on the Wi-Fi network at Levi’s Stadium, according to statistics provided by the Niners’ network team.

The Sept. 12 Monday Night Football game, a 28-0 win for the Niners over the visiting Los Angeles Rams, was well below the 10.1-TB mark recorded during Super Bowl 50, held at Levi’s Stadium back in February. Still, the 2.76 TB is a healthy regular-season game mark, with 16,681 unique users of the Wi-Fi network as well as a maximum concurrent number of users of 11,987.

The Niners also added some interesting new social twists to the Levi’s Stadium experience this year, including the ability for fans to use the stadium app to respond to poll questions (like voting on the next song to be played during timeouts) posted on the stadium’s large digital displays. While it’s not known if the feature was used during the regular season opener, according to the network team 30,000 fans participated in big-board polls during the Niners’ preseason game on Aug. 26 versus the Green Bay Packers. Full stats from the Niners networking team below.

Screen Shot 2016-09-23 at 12.01.42 PM

Yankee Stadium offers food ordering and delivery via VenueNext app

Home screen for VenueNext app for Yankee Stadium. Photo: Paul Kapustka, MSR

Home screen for VenueNext app for Yankee Stadium. Photo: Paul Kapustka, MSR

Fans in some areas of Yankee Stadium this year can now order food and beverages for in-seat delivery, thanks to a new stadium app developed with technology from VenueNext, the app developer behind the San Francisco 49ers’ Levi’s Stadium app.

Though the app isn’t part of the Major League Baseball official and approved game-day and stadium apps, it does offer most of the bells and whistles VenueNext developed for the Levi’s Stadium app, including digital ticketing, live wayfinding maps and public transit information. According to John Paul, the CEO of VenueNext, the food ordering option is now available to approximately 10,000 seats in the 54,251-seat Yankee Stadium, home of the New York Yankees and also the home to Major League Soccer’s New York City Football Club, which also uses the new app.

The VenueNext app comes courtesy of a deal struck last year between Legends Hospitality and VenueNext, to use VenueNext app technology at Yankee Stadium and at AT&T Stadium, home of the Dallas Cowboys. The Yankee Stadium app from Legends is the third major-league sport to use VenueNext technology to support in-seat food and beverage delivery, following the Niners’ app at Levi’s Stadium and an app for the Orlando Magic at Amway Center that debuted during the present NBA season.

App page showing in-seat food ordering and delivery option

App page showing in-seat food ordering and delivery option

In a phone interview with VenueNext’s Paul, he said that in Orlando the Magic started out with limited in-seat delivery, ramping up to offering it in the full lower bowl of Amway Center by the end of the regular season. According to Paul, the Yankees are using Aruba beacons to facilitate the wayfinding feature of the VenueNext app maps, and are using VenueNext’s Kezar ticket scanners to support digital ticketing. The Yankee Stadium app, however, does not yet support the ability to order food for express pickup at concession stands, Paul said.

No official word on Wi-Fi or MLBAM apps

The emergence of a VenueNext app that delivers capabilities not found in the so-called Official Yankee Stadium App raises some questions about whether or not the Yankees are playing ball with Major League Baseball Advanced Media’s strategy of having one single app for every MLB ballpark. MLB’s Ballpark app, for example, at Yankee Stadium offers “mobile check-in, social media, offers, rewards and exclusive content,” according to MLB. That’s a little bit different than the version of At Bat offered for the San Francisco Giants, which offers mobile ticketing support, seat upgrade options, and mobile food ordering. Other versions of Ballpark, for example for the Chicago Cubs and the Washington Nationals, offer fewer options. But as far as we know, there are no other MLB teams with a companion app like the VenueNext app for Yankee Stadium.

For both the Yankees and the Giants and all other teams, the MLB’s At Bat app offers live MLB content for a fee.

Yankee Stadium stadium map in the app

Yankee Stadium stadium map in the app

There is also no link to the new VenueNext app from the Yankees’ team website, and the VenueNext app does not contain any live content or replay options, features found on both the Niners’ and Magic’s apps from VenueNext. The Yankees have not yet replied to requests for information about the app and whether or not there is any public-facing Wi-Fi yet in Yankee Stadium.

Though MLBAM spent some $300 million last year to bring Wi-Fi and cellular DAS deployments to all MLB stadiums, Yankee Stadium was never confirmed to have had public Wi-Fi installed. Repeated requests to MLBAM asking about the Wi-Fi situation at Yankee Stadium have also not been returned.

NEW! Stadium Tech Report Podcast, Episode 1: What does Super Bowl 50’s Wi-Fi record mean for stadium tech pros?

Welcome to the inaugural episode of the STADIUM TECH REPORT PODCAST, with Mobile Sports Report editor Paul Kapustka and host Phil Harvey. In this first show Phil and Paul talk about the Wi-Fi and DAS records set at the recent Super Bowl 50 at Levi’s Stadium, exploring what those numbers mean for stadium tech professionals who are deploying their own networks — and whether or not there will ever be an end to the continuing explosive growth in demand for in-venue bandwidth.

Take a listen and let us know what you think in the comments!

Minnesota Vikings pick VenueNext for U.S. Bank Stadium app

Outside view of U.S. Bank Stadium in Minneapolis. Photo: USBankStadium.com.

Outside view of U.S. Bank Stadium in Minneapolis. Photo: USBankStadium.com.

Stadium app developer VenueNext has scored another NFL client, as the Minnesota Vikings announced today that they would use VenueNext technology in the app for the yet-to-open U.S. Bank Stadium.

According to VenueNext and the Vikings, the U.S. Bank Stadium app will support many of the same unique game-day features found in the app VenueNext built for the San Francisco 49ers’ Levi’s Stadium, including beacon-based wayfinding, the ability to order food and drinks via the app for express pickup, digital ticketing and game-day upgrade availability, as well as “robust” video content and a loyalty program tied to game-day activity. One feature at Levi’s Stadium, the ability to have food and drink delivered to fans in their seats, is “still being explored” by the Vikings, according to VenueNext.

Due to open this summer ahead of the 2016 NFL season, U.S. Bank Stadium is slated to host Super Bowl LII on Feb. 4, 2018. A Wi-Fi network with approximately 1,300 Cisco access points will supply wireless connectivity to the 66,200-seat venue, along with a neutral-host DAS built by Verizon Wireless. Aruba is supplying the 2,000 beacons being used inside the venue, and overall network operations will be run by CenturyLink, which will oversee deployment of some 2,000 digital TV displays inside the stadium.

Screenshot of U.S. Bank Vikings app in development. Image: VenueNext

Screenshot of U.S. Bank Vikings app in development. Image: VenueNext

According to VenueNext, app development partners will include Ticketmaster, Aramark for food, point-of-sale solution Appetize, seat upgrade technology from Experience, fan loyalty programs from Skidata and content app developer Adept. The Vikings are the third NFL team to choose VenueNext technology, behind the Niners and the Dallas Cowboys. VenueNext also has built a stadium app for the NBA’s Orlando Magic.

“We look forward to launching this new, dynamically-upgraded app that not only will give all Vikings fans a better experience when consuming team content on their mobile devices but also will allow seamless access to the numerous amenities at U.S. Bank Stadium,” said Vikings Owner/President Mark Wilf in a prepared statement. “Our goals are always to provide the best game day experience possible and to continue developing deeper engagement with all Vikings fans, and the VenueNext technology will help achieve both.”

“We’re excited to extend our reach in the NFL through this collaboration with the Vikings,” said John Paul, CEO & Founder of VenueNext, also in a prepared statement. “We want to become the standard for bringing Silicon Valley innovation to fan experiences, and implementing in a state-of-the-art development like U.S. Bank Stadium brings us closer to that goal.”

Interior look at U.S. Bank Stadium. Photo: USBankStadium.com

Interior look at U.S. Bank Stadium. Photo: USBankStadium.com

Betting the Under (Part 2): Putting Wi-Fi antennas under seats is the hot new trend in stadium wireless networks

Under-seat Wi-Fi AP at Levi's Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Under-seat Wi-Fi AP at Levi’s Stadium. Photo: Paul Kapustka, MSR (click on any photo for a larger image)

Part 2 of this story picks up with the decision to put Wi-Fi APs under seats at Levi’s Stadium. If you missed it, here is the link to Part 1.

According to Chuck Lukaszewski, now vice president of wireless strategy and standards at Hewlett Packard Enterprise (formerly very high density architect in the CTO Office of Aruba Networks), Aruba had been testing under-seat AP designs since around 2010, “in one form or another.” There were some initial tests of under-seat AP deployments at Turner Field in Atlanta and at American Airlines Arena in Dallas, but nothing on the scale of AT&T Park’s 2013 deployment, or on the scale Aruba planned to have at Levi’s Stadium when it opened in 2014.

Some of the first under-seat Wi-Fi deployments in other arenas were actually deployed completely under the stands, Lukaszewski said, with signals shooting up through the concrete. Though he said “you could get reasonably good throughput through concrete,” especially for 2.4 GHz frequencies, installing antennas above the concrete was “considerably better,” Lukaszewski said.

Curiously, one of the biggest problems in stadium Wi-Fi deployment — especially for those heavy on overhead antenna use — is negotiating interference between antennas; sometimes, clients can “see” antennas and APs that are across the stadium, and will try to connect to those instead of the AP closest to them, a problem that leads to inefficient bandwidth use. Interference also means you can’t place APs too closely together, making it somewhat of an art to find ways to increase coverage without increasing interference.

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi's Stadium. Photo: Paul Kapustka, MSR

Dan Williams, former VP of technology for the San Francisco 49ers, talking networking at Levi’s Stadium. Photo: Paul Kapustka, MSR

What Aruba found in its testing, Lukaszewski said, was that under-seat Wi-Fi AP deployments could be far more dense than overhead-centric designs, mainly because the human bodies in the seats would provide beneficial “blocking” of signals, allowing network designers to place APs more closely together, and to be able to re-use the same Wi-Fi channels in more antennas.

“If you can use human bodies to contain signals, you can have much smaller cells,” Lukaszewski said. Under-seat deployments, he said, “allows us to re-use the same channel less than 100 yards away.”

With more channels available for each AP, the difference in the metric Lukaszewski calls “megabytes per fan” can be “profound” for an under-seat design versus an overhead design, he said.

“We do see trends [in stadium network data] of under-seat being able to deliver well over 100 MB per fan per event, while overhead designs [deliver] significantly under 100 MB per fan per event,” said Lukaszewski.

Dan Williams, the former vice president of technology for the San Francisco 49ers, said he and Lukaszewski were in agreement that under-seat was the best method to deploy at Levi’s Stadium.

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

Kyle Field at Texas A&M. White spots in stands are under-seat AP locations. Photo: Paul Kapustka, MSR

“I just did not believe in overhead,” said Williams, who said he brainstormed with Aruba’s Lukaszewski on the under-seat idea, which they both brought to the Wi-Fi design at Levi’s. By using under-seat APs, Williams said, the Levi’s Stadium design looked to provide “cones [of bandwidth] around the audience, immersing [fans] in a signal.”

After beating the previous year’s Super Bowl Wi-Fi total at its NFL regular-season opener in 2014, Levi’s Stadium’s Wi-Fi network more than passed its biggest test ever this year, carrying a record 10.1 terabytes of Wi-Fi data during Super Bowl 50. Those numbers are proof of Lukaszewski’s claim: “By far, under seat is better.”

New deployments trending to under-seat

Editor’s note: This excerpt is from our latest STADIUM TECH REPORT, our long-form PDF publication that combines in-depth stadium tech reports with news and analysis of the hottest topics in the world of stadium and large public venue tech deployments. Enjoy this PART 1 of our lead feature, or DOWNLOAD THE REPORT and read the whole story right now!

Even though under-seat deployments can be considerably more expensive, especially in a retrofit situation where deployment requires coring through concrete, many stadiums are now seeming to agree with another Lukaszewski claim, that “the return absolutely justifies the investment.”

At AT&T Stadium in Arlington, Texas, the Cowboys quicked followed their sister park’s lead and installed under-seat APs in force ahead of that venue’s hosting of the inaugural College Football Playoff championship game in January of 2015. John Winborn, chief information officer for the Dallas Cowboys Football Club, said the team worked with AT&T’s “Foundry” innovation centers to produce a smaller, sleeker under-seat AP enclosure that fit well with the stadium’s commitment to aesthetics.

Back on the baseball side, the Giants now have 1,628 Wi-Fi APs in their park, with the vast majority of them under-seat, in all three decks of seating. And the Giants’ main rival to the south, the Los Angeles Dodgers, also used under-seat APs in a recent Wi-Fi upgrade.

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

Close-up of conduit running to under-seat AP at Kyle Field. Photo: Paul Kapustka, MSR

And if Levi’s Stadium led the way for under-seat Wi-Fi, the new mainly under-seat network at the refurbished Kyle Field at Texas A&M might be the QED on the debate, with ultra-fast network speeds and big data-consumption numbers (including 5.7 TB of Wi-Fi at a game versus Alabama) adding measureable momentum to the under-seat trend. Bill Anderson, CEO of Wi-Fi deployment strategy firm AmpThink, said he was an early disbeliever in under-seat Wi-Fi — until he saw the numbers.

“At first we mocked it, made fun of it,” said Anderson, whose firm has been called in to produce Wi-Fi network designs for several recent Super Bowls, as well as for the Kyle Field design. But when Aruba showed AmpThink the data from under-seat tests and deployments, “that was the ‘a-ha’ moment for us,” Anderson said.

Working with Aruba at Kyle Field, AmpThink was able to collect its own data, which convinced Anderson that under-seat was the way to go if you wanted dense, high-performing networks.

“The really important thing is to get APs closer to the people,” said Anderson. “That’s the future.”

Anderson said some doubters may remain, especially those who try to mix a small amount of under-seat APs with existing overhead deployments, a recipe for lowered success due to the potential interference issues. At Texas A&M, Anderson said AmpThink was able to build a design with far less interference and much greater density than an overhead solution, producing numbers that people have to pay attention to.

“We only know what we’ve observed, but we’re evangelistic supporters” of under-seat designs, Anderson said. “If someone says to you under-seat is hocus-pocus, they’re not looking at the data.”

Not for everyone, but more are trying under-seat

Though proponents of under-seat Wi-Fi all agree on its ability to deliver denser, faster networks, they all also agree that under-seat can be considerably more costly than overhead Wi-Fi, especially in a retrofit situation.

In addition to having to core through concrete seating areas to get conduit to the under-seat APs, the devices themselves need to be sealed, to guard them from weather, drink spills, and the power-washing equipment employed by most stadiums to clean seating areas.

Aruba’s Lukaszewski also noted that under-seat deployments generally use more linear feet of cabling to connect the APs than overhead, which also drives up the cost. Then since under-seat designs tend to use more APs, that also means a higher budget to cover a higher number of devices.

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

A row shot of the under-seat APs at AT&T Stadium. Photo: Dallas Cowboys

For some stadiums, the construction materials used prohibit the under-seat option from even being tried. At the Green Bay Packers’ legendary Lambeau Field, a late-1950s construction design that used lots of concrete and rebar — as well as part of the stadium’s bottom sitting directly in the ground — meant that under-seat Wi-Fi wasn’t an option, according to Wayne Wichlacz, director of information technology for the Packers.

Other stadiums, like the University of Nebraska’s Memorial Stadium, don’t have enough space between the stadium’s bleacher seats and the floor for under-seat APs to be safely installed. And many schools or teams simply don’t have big IT budgets like the $20-million-plus available to Texas A&M that allowed the Kyle Field design to seek the best result possible.

But many of the new stadiums under construction, as well as existing venues that are planning for new best-of-breed networks, have already committed to under-seat Wi-Fi designs, including the Sacramento Kings’ Golden 1 Center, where Ruckus Wireless will implement its first under-seat stadium Wi-Fi network.

Steve Martin, senior vice president and general manager at Ruckus, said the Golden 1 Center design, planned to be the most dense anywhere, will “primarily be underseat,” a choice he said “helps in a lot of ways.”

Foremost is the performance, something Martin said Ruckus has been testing at the Kings’ current home, the Sleep Train Arena. “It [under seat] does give you the isolation for frequency re-use,” he said.

The under-seat design also makes sense in Golden 1 Center since the stadium’s overall design is very open, with lots of glass walls and unobstructed views.

And under-seat deployment is even making inroads into the distributed antenna system (DAS) world, with Verizon Wireless implementing more than 50 under-seat DAS antennas at Levi’s Stadium prior to Super Bowl 50. Mainly installed to cover the bottom-of-the-bowl rows, the under-seat APs helped Verizon manage a record day for DAS traffic, with 7 TB reported on its in-stadium cellular network during the game.

“To get a quality signal, we had to go under seat,” said Brian Mecum, vice president, network, for Verizon Wireless, who said that in that area of the stadium, under seat was the only way to get a quality signal close to the subscriber’s phone. Verizon, he said, helped design the under-seat DAS antenna, and is looking to deploy it in other stadiums soon.

“It’s the first of more,” he said.

END PART 2… HERE IS THE LINK TO PART 1… TO READ THE WHOLE STORY NOW, DOWNLOAD OUR REPORT!