The development of our Dell EMC PowerScale and ECS platforms was informed by the challenges the enterprise faces when scaling distributed systems like Hadoop. As data teams continue to scale their Hadoop and analytics systems, the need increases for flexible compute and storage. Data teams now are processing more data than ever before, but with the growth of data comes significant management challenges. To address these issues, many data teams pivot to architectures that allow for independent scaling of compute and storage in both Object and HDFS for Hadoop. At Dell Technologies, we have helped our customers work through these challenges for many years.

Since collaboration with Hortonworks and Cloudera began in 2015, Dell Technologies has engaged in joint engineering and validation efforts to bring our leading edge file and native HDFS storage product Dell EMC PowerScale and distributed object storage product Dell EMC ECS to both Hortonworks Data Platform (HDP) and Cloudera Data Hub (CDH).

Extending the Partnership

With the release of the Cloudera Data Platform (CDP), the Cloudera team is enabling IT to deliver easier, faster, and safer self-services analytics experiences. Today, we are announcing that we will work with Cloudera to validate and certify CDP with PowerScale OneFS and ECS. Our new partnership is built on the base of many years of QATS certification for both CDH & HDP platforms with our unstructured data solutions.

“As customers continue to expand their Machine Learning workloads and the storage requirements evolve, we’re excited to collaborate with Dell Technologies to bring to market solutions backed by its leading-edge unstructured data storage offerings like PowerScale and ECS,” said Nadeem Asghar, VP of Solutions and Partner Engineering at Cloudera. “Dell Technologies shares our commitment to ensuring our customers can always stay ahead of industry and technology trends and we look forward to delivering solutions to our customers for years to come.” 

 Benefits for Data Teams

This new three-year investment strengthens the Dell Technologies and Cloudera relationship, allowing us to:

1. Continue to support our existing joint customers on existing and future hardware and software releases.
2. Bring shared storage model at scale with innovative and fully validated end-to-end platforms to support the growing Hadoop ecosystem.

Over the course of next few months, we are contracted to work jointly with Cloudera to certify PowerScale as the primary HDFS store for CDP-Private Cloud Base 7.1.x. In the same timeframe, we also plan to certify Dell ECS through QATS as the S3 object store for CDP 7.1.x.

Building a Solid Data Foundation for Analytics

Finally, PowerScale’s capability for data consolidation that can manage data for several Hadoop distributions simultaneously enables us to offer phased migration services from CDH or HDP to CDP. This simplifies the process and significantly minimizes business risk in migrating to the new Hadoop distribution. At Dell Technologies, we plan to launch these migration services as CDP-Private Cloud Base becomes available for on-prem deployment.

You can find more information about Data Analytics and Hadoop solutions built using PowerScale  here and Apache Spark on PowerScale here. For the technically inclined, you can find technical details on Hadoop with PowerScale here.