There are seven words you can’t say on television, but only one that would-be viewers hate to see fill their online video screens: buffering. Researcher NScreenMedia estimates media companies lose $2 billion a year in ad revenue because 1 in 5 consumers gives up on patchy online videos. Computer science professor Hui Zhang says he can stop that.

Zhang’s San Mateo (Calif.) startup, Conviva, makes stream-monitoring software that big companies can use to ease network congestion. Able to oversee millions of connections at a time—as it did during its first big test, the 2008 Olympics—the software determines whether incoming video is lagging what’s on the screen. If it’s slower, Conviva compensates by lowering the resolution or switching the viewer to a different server. While companies monitoring network capacity typically keep tabs only on their own server loads, “If you have a real-time sense of what’s happening with each video player, there are ways to infer what is happening inside the network and decide what optimization steps to take,” says Zhang, 45, who teaches at Carnegie Mellon University in Pittsburgh.

Born in Harbin in northeastern China, Zhang enrolled in a science and technology university at 16. He studied computer science on his high school math teacher’s advice. In 1989, a year after starting doctoral studies at University of California at Berkeley, Zhang joined a group researching how to improve online video quality. “It was clear the killer app was going to be video,” he says, and U.S. technology was in the lead. His research on peer-to-peer streaming, including a test that brought clear video to thousands of users at once, planted the seeds for Conviva, started in 2006 as Rinera Networks. Staffers pulled some all-nighters to fix problems with software Conviva sold to its first clients, Yahoo! (YHOO) and Italy’s KataWeb, but hasn’t had such issues since, Zhang says.

Media companies including HBO (TWX), NBCUniversal (CMCSA), and Yahoo! have been using Zhang’s software in their video streaming applications for years, some since 2007. His company got its first big break when Zhang convinced former NBCUniversal Chief Technology Officer Darren Feher that he’d need Conviva’s live monitoring systems to manage streams of the Beijing Olympics, which totaled about 100 million over two weeks. In 2009, Feher signed on as Conviva’s chief executive officer. The company—which has raised $59 million in venture capital since 2006, including a recent $15 million led by Time Warner Investments—has 75 employees and revenue in the “double-digit millions,” Feher says, declining to elaborate.

He adds that Conviva, which now oversees 3 billion video streams monthly, negotiates fees differently with each client. Colin Dixon, chief analyst for NScreenMedia, says Conviva’s strength lies in adding viewer analytics like those of Adobe (ADBE) or Ooyala to content delivery akin to Akamai (AKAM). It has no direct competitors, he says. “It’s really tough to process and deliver video at the same time.”

Problem

Twenty percent of streaming video viewers abandon slow feeds

Solution

Software shuffles servers and tweaks resolution

Who’s buying

Clients include HBO, NBCUniversal, and Yahoo!

Read to the story

There are seven words you can’t say on television, but only one that would-be viewers hate to see fill their online video screens: buffering. Researcher NScreenMedia estimates media companies lose $2 billion a year in ad revenue because 1 in 5 consumers gives up on patchy online videos. Computer science professor Hui Zhang says he can stop that.

Zhang’s San Mateo (Calif.) startup, Conviva, makes stream-monitoring software that big companies can use to ease network congestion. Able to oversee millions of connections at a time—as it did during its first big test, the 2008 Olympics—the software determines whether incoming video is lagging what’s on the screen. If it’s slower, Conviva compensates by lowering the resolution or switching the viewer to a different server. While companies monitoring network capacity typically keep tabs only on their own server loads, “If you have a real-time sense of what’s happening with each video player, there are ways to infer what is happening inside the network and decide what optimization steps to take,” says Zhang, 45, who teaches at Carnegie Mellon University in Pittsburgh.

Born in Harbin in northeastern China, Zhang enrolled in a science and technology university at 16. He studied computer science on his high school math teacher’s advice. In 1989, a year after starting doctoral studies at University of California at Berkeley, Zhang joined a group researching how to improve online video quality. “It was clear the killer app was going to be video,” he says, and U.S. technology was in the lead. His research on peer-to-peer streaming, including a test that brought clear video to thousands of users at once, planted the seeds for Conviva, started in 2006 as Rinera Networks. Staffers pulled some all-nighters to fix problems with software Conviva sold to its first clients, Yahoo! (YHOO) and Italy’s KataWeb, but hasn’t had such issues since, Zhang says.

Media companies including HBO (TWX), NBCUniversal (CMCSA), and Yahoo! have been using Zhang’s software in their video streaming applications for years, some since 2007. His company got its first big break when Zhang convinced former NBCUniversal Chief Technology Officer Darren Feher that he’d need Conviva’s live monitoring systems to manage streams of the Beijing Olympics, which totaled about 100 million over two weeks. In 2009, Feher signed on as Conviva’s chief executive officer. The company—which has raised $59 million in venture capital since 2006, including a recent $15 million led by Time Warner Investments—has 75 employees and revenue in the “double-digit millions,” Feher says, declining to elaborate.

He adds that Conviva, which now oversees 3 billion video streams monthly, negotiates fees differently with each client. Colin Dixon, chief analyst for NScreenMedia, says Conviva’s strength lies in adding viewer analytics like those of Adobe (ADBE) or Ooyala to content delivery akin to Akamai (AKAM). It has no direct competitors, he says. “It’s really tough to process and deliver video at the same time.”

Problem

Twenty percent of streaming video viewers abandon slow feeds

Solution

Software shuffles servers and tweaks resolution

Who’s buying

Clients include HBO, NBCUniversal, and Yahoo!

Read the story on Businessweek

Conviva monitors and optimizes tens of millions of online video streams daily for premium video brands. Through Conviva Pulse, our online video dashboard, customers analyze how their online video is being consumed. For example, customers can in real-time identify the most popular videos being watched and adjust their advertising strategy. As another example, customer ops teams can in real-time detect problems degrading the experience of users watching a live basketball game (say, high buffering due to an overloaded CDN) and quickly take corrective action before the game ends. In addition to live monitoring, our customers can also analyze historical video trends – what were the most popular videos last week and how long were they watched on average?

Our customers also ask questions that require very deep and often ad-hoc analysis – “Something seems to be wrong with my video delivery last week. Any idea what is going on?”  The video analysis team at Conviva digs through terabytes of data to provide detailed responses to such questions. This post describes how we use Hive and Spark to make this happen.

First, here is an overview of how data flows through the Conviva infrastructure. Conviva monitoring and optimization code embedded within our customers’ video players collects non-personally identifiable information like buffering time and frame rate, and sends it to Conviva’s backend infrastructure every few seconds. The data entering our backend is replicated into two streams. One stream goes into our custom live data processing stack written in Java. This live system solves the challenging task of analyzing multiple millions of concurrent video streams with a 1 second latency, and deserves its own separate blog post. The other stream is written into our Hadoop File System (HDFS) cluster. MapReduce jobs summarize this raw data and extract out the historical trending information shown in Pulse. The raw data written to HDFS, together with the summaries calculated by the MapReduce jobs, constitute the input to our Hive and Spark based offline reporting and ad-hoc analysis infrastructure, which is described next.

We first started with Hive. Hive provides a SQL-like interface on top of data stored in HDFS. For example, we can find the number of sessions per video on a particular day with the following Hive query:

SELECT videoName, COUNT(1)
FROM summaries
WHERE date='2011_12_12' AND customer='XYZ'
GROUP BY videoName;

Our analysis team uses Hive queries to dig deep into the data and answer customer questions. This process often involves tens of ad-hoc queries, most of them filling up multiple A4-sheets of paper. The most commonly used queries are packaged up together into reports on our internal analysis web-portal so that engineers and customer service representatives can easily run them. Our analysis team also uses Hive queries to measure and improve the performance of our video optimization algorithms, and to gain value insights into the online video ecosystem and user behavior. We have published some of our findings in the research paper titled “Understanding the Impact of Video Quality on User Engagement”, published at SIGCOMM 2011.

Hive works great for performing ad-hoc queries against large amounts of data stored in HDFS. The only problem is that queries often take a long time, as they must read data from HDFS. Most of the time, we run multiple queries against the same data set. Hive re-reads the data from HDFS each time.  This is highly wasteful and slow. This is where Spark helps.

Spark is an in-memory distributed computing framework being developed at UC Berkeley. Spark allows us to load the data of interest from HDFS (or any persistent storage) into RAM across multiple servers and cache it. We can then perform multiple queries against the cached data. Since the data is in RAM, queries are super quick. If a node dies, Spark automatically reconstructs the data from persistent storage.

Queries in Spark are not SQL-like queries as in Hive. Instead, they are written in Scala. The following Spark query produces the same results as the video count Hive query we saw earlier.

val sessions = sparkContext.sequenceFile[SessionSummary, NullWritable](pathToSessionSummaryOnHdfs, 
                   classOf[SessionSummary], classOf[NullWritable]).flatMap {
                       case (key, val) => val.fieldsOfInterest
               }
val cachedSessions = sessions.filter(whereConditionToFilterSessionsForTheDesiredDay).cache
val mapFn : SessionSummary => (String, Long) = { s => (s.videoName, 1) }
val reduceFn : (Long, Long) => Long = { (a,b) => a+b }
val results = sessionsForQuery.map(mapFn).reduceByKey(reduceFn).collectAsMap

The above Scala code first reads the session summaries from HDFS. We only keep the subset of fields that are necessary for the query in order to conserve memory. We then filter out sessions that do not match the day and customer of interest. We now have our working set, and instruct Spark to cache it. After that, we start up our group by video name query which is defined as a map-reduce job.  At this point, Spark reads the data from HDFS, caches it, executes the map-reduce job and produces the results.

If we run just one query, Spark is not much faster than Hive. Reading the data from HDFS is the bottleneck in both cases. Most of the time, however, we have to issue multiple queries on the same data – say, we want to find the video counts per country, state and city in addition to the video name.  In that scenario, Spark is much faster than Hive. Since the data is already cached in memory, subsequent queries complete very quickly. One of our reports which produces detailed video data analysis for a geographic region took 24 hours to run using Hive. After converting to Spark, the same report runs in under 45 minutes. The tremendous savings produced by Spark have helped us run more reports than we could have with our existing Hadoop cluster. Today, over 30% of the reports we run on a daily basis use Spark.

For simple queries, writing the query in Spark is harder than writing it in Hive. However, it is much easier to write complicated queries in Spark than in Hive. Most real-world queries tend to be quite complex; hence the benefit of Spark. We can leverage the full power of the Scala programming language, rather than relying on the limited syntax offered by SQL. For example, the Hive expression IF(os=1, “Windows”, IF(os=2, “OSX”, IF(os=3, “Linux”, “Unknown”))) can be replaced by a simple match clause in Scala. You can also use any Java/Scala library to transform the data. Moreover, an ongoing project at UC Berkeley called Shark, will soon make it possible to run Hive queries on top of Spark.

Overall, Spark has been a great boon for data processing operations at Conviva. We are currently working on leveraging Spark in many more ways, including real-time analysis and anomaly detection. If you are interested in working on large-scale data processing systems,we are hiring.