Transforming Storing – and Processing – of Video Data
- February 10, 2021
Innovation in technology changes more than just the tools organizations use. It also fundamentally changes how organizations operate. This is quite evident for companies working with video. The exponential rise in data exacerbates throughput, efficiency, scalability, and reliability issues. To stay profitable, they must update their equipment as well as transform their internal work processes.
Consider the difficulties faced by EarthCam, a provider of webcam technology and managed services consisting of streaming video, time-lapse movies, and 360° reality capture. Every month, the company brings millions of high-resolution images into their network. To date, their database contains over a billion files ranging from one to 24 MB in size. The vast number of files created complexity in their storage system. One way to manage different types of data is to put them in different silos. While this makes it easier to manage each type of data individually, it complicates overall storage management because each silo needs to be managed differently in terms of factors such as RAIDs, RAID groups, and protocols. Scaling was also challenging because each addition to the network increased the complexity of managing the overall storage array.
Working with stored data was also quite inefficient. For many of its clients, EarthCam creates time-lapse movies, such as compressing images of a three-month construction project into a one-minute video. Each image in the time-lapse movie needs to be hand-selected from the tremendous number of images available and then enhanced to provide a consistent look and feel. Traditionally, the entire image set is downloaded to the editor’s workstation. Downloading this much data puts a strain on available bandwidth and causes delays across all network workloads.
EarthCam also wanted to reduce its operating costs without compromising efficiency, quality, or reliability. And, with rising success came the need to easily scale storage while simplifying overall management of the network.
Increased Efficiency, Productivity, and Operational Savings
Dell EMC PowerScale storage, the next-generation scale-out storage of the Emmy-winning Isilon lineage, is designed for applications with data requirements ranging from terabytes up to 10 petabytes – the right platform for EarthCam. Typical storage utilization tops out at 66%. The advanced capabilities of PowerScale – including globally coherent cache, load-balancing, and automated tiering – increase performance efficiency for every application and deliver 80%+ utilization.
EarthCam’s new system did more than just storing data. By adding a cluster of fast-performing PowerScale nodes, EarthCam was able to shift their video-editing platform to become local with image data. Now, instead of downloading an entire timeline of images, editors work with just the data they need to complete their job.
For example, for a time-lapse movie, editors can select the images to use from thumbnails rather than hi-res originals. These thumbnails are created on processing resources local to the stored images, reducing how much data needs to be downloaded by orders of magnitude. This in turn leads to direct operational savings, both in terms of bandwidth and the time it takes to complete a task. For EarthCam, working locally reduced processing time for the video production team by 20%.
The savings did not stop there. PowerScale doubled EarthCam’s storage capacity while actually reducing the rack space needed. Overall, they were able to achieve about 26X greater density and 100X faster sustained throughput compared to their previous storage implementation.
Seamless Scalability and Ease of Use
Another important requirement for EarthCam was simplified network management. Typical storage scaling is achieved by adding “bricks” of storage. The problem with this approach is that LUNs need to be set up and each node needs to be managed. With PowerScale, IT plugs in a new node and, without needing to reconfigure the data pool, data is immediately migrated over to the new node to balance the load across the entire array and maximize performance. Adding a node in this way is relatively seamless and can be completed with no downtime.
EarthCam was able to simplify day-to-day management by consolidating petabytes of different types of data in one location, eliminating the need for data silos and managing different storage systems. Single volume / single file system / single namespace consolidation both on-premises and in the cloud greatly simplifies workflows. Eliminating silos also enables greater collaboration between groups who can now share collaborative tools and data assets.
PowerScale InsightIQ and other tools provide greater visibility into storage processes and network loads. With such visibility, IT doesn’t have to work down into the stack to manage the network. For example, EarthCam can manage their network, such as changing performance on directories based on usage, without having to worry about specific LUNs, and such changes take just minutes to apply. In short, a single person can manage petabytes of different types of data. This frees up IT personnel from managing data storage, enabling them to focus on optimizing applications and how data can be used more efficiently.
Upgrading to next-generation PowerScale technologies has been seamless as well. After plugging in new PowerScale nodes, the OS can be configured to begin evacuating data from nodes to be retired into the new nodes. The entire migration can take place with zero downtime.
Ease of use has translated directly to the bottom line for EarthCam: since implementing PowerScale, they have had an 82% reduction in systems administration resources and been able to maintain five nines (99.999%) uptime.
Transforming a business doesn’t have to be hard. The fact is, simplifying transformation is the core job of technology companies. At Dell Technologies, we understand that increasing the efficiency of business operations is just the first step. Part of bringing innovation to market is helping facilitate the business transformations that are essential to the process of growth.