Delivering a Flexible IT Infrastructure for Analytics on IBM Power Systems

Editor's Notes

1. TALK TRACK This presentation provides an overview of the Hortonworks and IBM Power partnership and how it will enable clients rapid access to open innovation and drive key business value on their analytics journey.
2. TALK TRACK 1st I will review the partnership announce at IBM Edge in Sept 2016 2nd I will look at an example of a client’s planned journey with HDP and Power Systems and the renovate to innovate philosophy Next I will review how Hortonworks leads the open community innovation for Hadoop and supports their client with business success Lastly I will overview how OpenPower innovation delivers outstanding time to insights for analytics
3. TALK TRACK Overview the recently announced partnership between IBM and Hortonworks that will bring open Hadoop to OpenPOWER Encourage clients to view the videos of Scott Gnau talking about Hortonworks perspective and James Wade who talked about how Florida Blue is leveraging open source HDP and OpenPower to power his analytics journey.
4. TALK TRACK Refer to the linked blog for the full background: http://hortonworks.com/blog/delivering-flexible-infrastructure-analytics-hortonworks-hdp-ibm-power-systems/ Summary of Florida Blue’s three new business models (expanding from one to four businesses): started with a B2B insurance model, entering retail insurance, now handling 32% of US government Medicare claims and opening provider clinics in Florida Florida Blue has seen outstanding 3-5x performance results of running MongoDB on IBM OpenPOWER servers to support their call center initiatives. Note that they are using both Hortonworks and IBM Power today and are planning to bring these capabilities together as one of the 1st adopters of the HDP 2.5 for Linux on Power within the early client program.
5. TALK TRACK At Hortonworks, we partner with our customers and guide them on their Journey to Actionable Intelligence. You can start your journey anywhere you want. You can renovate your IT architecture to reduce costs and boost functionality. Or you can innovate modern data applications that you use at your own company or sell on the open market. You can start with the most sophisticated use cases if your team is experienced, or you can build your expertise by beginning with less complex use cases that bring quick results. As you build your team’s expertise and comfort with Hortonworks Data Platform and Hortonworks DataFlow, you can then tackle more challenging aspects of your road map. We will help you plan the right path to meet your objectives. Now I’d like to tell you about some Hortonworks customers and how they are transforming their industries with the actionable intelligence that comes from Connected Data Platforms. [NEXT SLIDE] DISCUSSION STRATEGY [For Business Prospects] Focus questions What business problems can we help you solve? Which use case would you like to tackle first? What type of challenge is most important: data discovery, building a single view or creating predictive analytics? Calls to action Recommend the Jumpstart package: http://hortonworks.com/products/subscriptions/jumpstart/ Schedule a use case workshop and plan your journey across your most important use cases. Give them an industry-specific White Paper to read. [For IT Prospects] Focus questions Where do you face the most cost pressure to store and process data? Which use case would you like to tackle first: active archive, ETL offload or data enrichment? Calls to Action Recommend that they download Hortonworks Sandbox: http://hortonworks.com/products/hortonworks-sandbox/ Schedule a use case workshop Give them the EDW Optimization White Paper to read: http://hortonworks.com/solutions/edw-optimization/
6. TALK TRACK When you choose a vendor, you choose a platform that will be with you for a long time. As the originators of both the Apache Hadoop and Apache NiFi technologies as well as the Open Enterprise Hadoop category, Hortonworks is uniquely positioned to help you transform your business with actionable intelligence. We are the only publicly traded pure-play Hadoop company Our momentum is accelerating. Our customers come to us for all the reasons I’ve described: our superior technology that evolves at the pace of open innovation, our proven model for partnering with our customers and our dedication to helping them succeed. I’d like to come back for a larger meeting and a use case workshop to start the journey together. Would next week work for you? [NEXT SLIDE] SOURCE: http://hortonworks.com/about-us/quick-facts/
7. TALK TRACK Hortonworks has always followed a 100-percent open approach to Hadoop and Hortonworks Data Platform is the only genuinely open Hadoop distribution. Our open approach: Eliminates risk of becoming locked in with one proprietary vendors It maximizes innovation by tapping into the ingenuity of the largest number of talented developers with the broadest perspective of the capabilities and areas for improvement It integrates seamlessly with other datacenter technologies. Like our customers, our technology partners also want to avoid getting locked in to a proprietary approach. [NEXT SLIDE] SUPPORTING DETAIL Eliminates Risk Why this matters to our customers: Their interests in Hadoop align with Hortonworks’ incentives to deliver world-class support and constantly improve Hadoop Proof point: Spotify was running Cloudera without support. As the importance of Hadoop grew in their business, they needed support. They chose to migrate from CDH AND to begin paying Hortonworks for support, instead of staying on CDH unsupported Citation: "[Hortonworks'] true open source approach and the work they have done to improve the Apache Hive data warehouse system aligns well with our needs," said Wouter de Bie, team lead for data infrastructure at Spotify, in a statement. "We use Hive extensively for ad-hoc queries and for the analysis of large data sets.” | http://www.informationweek.com/software/information-management/spotify-embraces-hortonworks-dumps-cloudera/d/d-id/1111563? Maximizes Community Innovation Why this matters to our customers: They benefit from the fastest possible innovation that comes from the real-world needs felt by our HDP subscribers Proof point: The Stinger Initiative organized the efforts of 144 developers at 45 companies to write 390,000 lines of java code in 13 months. This combined effort made the most important Hive queries 100X faster. Citation: “Combining the capabilities of HARMAN services and Hortonworks, automakers and their suppliers will have access to a scalable platform for, real-time insights, new innovative service creation and predictive analytics-based solutions that can minimize the risk of costly recalls and reduce warranty expenses,” said Sanjay Dhawan, President, HARMAN Services Division. “This is a very exciting step forward in the evolution of the connected car as we speed up the time to market with powerful functionalities that benefit the OEMs and their drivers.” Integrates seamlessly Why this matters to our customers: Rapid, easy adoption that reinforces your existing datacenter technologies Proof point: More than 130 technology partners are HDP-certified, including: EMC, HP, Microsoft, Red Hat, SAP, Teradata, Cisco, SAS and Google Cloud Platform Citation: “The new features in the Hortonworks Data Platform 2.3 allow us to empower our joint customers with access to more data through Apache Hadoop,” said Randy Guard, Vice President of Product Management, SAS. “This allows them to perform deeper analysis on their data with extensive security and data governance capabilities in place. We want our customers to know that we support the platforms and environments they need to handle their data-driven workloads in a flexible, secure and efficient way.” | http://hortonworks.com/press-releases/hortonworks-accelerates-big-data-transformations-with-hortonworks-data-platform-2-3/
8. TALK TRACK Hortonworks employs the largest number of project committers to Apache Hadoop, Apache NiFi and their related Apache projects. Our committers innovate with other leaders working through the Apache Software Foundation. We have more than 90 Apache committers, and they interact with each other every day in meetings, in our hallways or over lunch, And our leadership in the Apache community also helps us influence the roadmap, to meet the needs that we learn about from our hundreds of enterprise subscribers. The Apache way is democratic, but we have the largest voting block. [NEXT SLIDE] SUPPORTING DETAIL We Employ the Committers Why this matters to our customers: No company understands Hadoop better than Hortonworks. This correlates directly to the quality of our enterprise support. Proof point: Specific Apache Hadoop committer numbers as of January 2016 (# of committer seats / % of total committers): Hadoop: Hortonworks = 30/31%, Cloudera = 17/71% Customer quote: Paul Boal, “Whenever our developers need access to some knowledge, and we say ‘Could we talk to somebody about that?’, Hortonworks says, ‘Sure, here are a few of our committers. Why don’t you talk to them?’“ | https://youtu.be/iQ2V3FuqxHg @ 3:04 Our Committers Innovate Why this matters: Choose the team that leads Hadoop innovation, so you can benefit from a rapid flow of new features that directly benefit your business. Proof points: YARN – YARN is the data operating system of Hadoop 2. Arun Murthy originally conceived of YARN in the JIRA ticket MapReduce 279. The founding architects of Hortonworks developed YARN until it was ready for release as part of Apache Hadoop 2.0, which was GA in October 2013. | http://hortonworks.com/blog/introducing-apache-hadoop-yarn/ The Stinger Initiative: In 2013, our customers told us that Hive was too slow and that it didn’t include all the semantics they needed. So we spearheaded the Stinger Initiative to improve the speed, semantics and scale of Hive queries. Over 13 months and three versions of Hive (0.11, 0.12 and 0.13) – 144 developers from 45 companies wrote 390 lines of java code. This improved Hive’s speed by more than 50x and added the missing SQL semantics. | http://hortonworks.com/blog/announcing-apache-hive-0-13-completion-stinger-initiative/ Partner quote: “Hortonworks has partnered with Red Hat since its early days. They have been a champion of the open source model and a key enabler of the communities needed to support the big data ecosystem,” said Greg Kleiman, director of big data strategy at Red Hat. “As a strategic partner, Hortonworks brought their flavor of open source innovation to build more agile and cost effective big data solutions with Red Hat.” | http://hortonworks.com/press-releases/hortonworks-recognized-as-crn-emerging-vendor/ We Influence the Roadmap Why this matters to our customers: They can influence the roadmap and attract top technical talent to join their teams Proof point: The Data Governance Initiative initiative led to the creation of Apache Ranger. Hortonworks encouraged its subscribers Schlumberger, Aetna, Target and Merck to join the initiative to speak for their requirements and to engage their teams to solve for those requirements. This led directly to creation of the Apache Atlas project, and each of these customers employs at least one committer to Atlas. | http://hortonworks.com/blog/apache-atlas-project-proposed-for-hadoop-governance/ Analyst Quote: “Applying old-school dictatorial-style governance to Hadoop would be a disaster for enterprises because it would tamp down the agility of Hadoop,” he said. “The Data Governance Initiative is intriguing because it brings the Hadoop community together with real enterprises. I hope they partner in a way that create ‘minimally viable governance.’ That would keep Hadoop flexible while giving enterprises the governance they need to prevent data pandemonium.” | http://www.bigdatatechcon.com/news/data-governance-initiative-expands-the-hadoop-ecosystem
9. TALK TRACK We know that the community powers the ongoing success of our technology, and so we provide a platform for that community through Hortonworks Community Connection. As a founding member of ODPI, we help influence the larger ecosystem that is standardizing around Apache Hadoop as they develop new Big Data applications. And we engage our partner ecosystem community through the Hortonworks Partnerworks program. [NEXT SLIDE]
10. TALK TRACK Hortonworks Support Subscriptions come with Hortonworks SmartSense to help you proactively and predictively optimize your HDP cluster. SmartSense analyzes the data on how you use your cluster, compares that to best practices and makes automated recommendations about how you can optimize your investment in HDP. We also offer a variety of self-service tools through our integrated customer portal. And of course, you can calls us 24x7. We’ll always pick up, and for any tough issues, our support engineers have access to our committers. [NEXT SLIDE] SUPPORTING DETAIL Hortonworks SmartSense Why this matters to our customers: Hortonworks SmartSense helps our customers make the most of their Hadoop cluster – using data. Proof point: Just like our customers use machine data for preventative maintenance on their telco infrastructures, oil rigs or military aircraft, they can use the same type predictive analytics using data from their own cluster to optimize that resource. In fact, Hortonworks subscribers benefit (through SmartSense) from the usage data from all other Hortonworks customers. Citation: “SmartSense’s capabilities, says Cheolho Minale, vice president of technology at The Mobile Majority will allow his Hadoop team to optimize it’s HDP cluster’s ad performance, ‘At The Mobile Majority, we have been using Hortonworks Data Platform to optimize ad performance on behalf of our customers. We’re excited to look into Hortonworks SmartSense as a way to continuously optimize our HDP cluster as it grows over time.’” | Source: http://hortonworks.com/blog/introducing-availability-of-hdp-2-3-part-3/ Integrated customer portal Why this matters to our customers: Hortonworks subscribers can scale their organization’s Hadoop expertise as quickly as they want.
11. TALK TRACK IBM Power Systems, with our OpenPOWER partners, offer a full range of server solutions from public cloud to distributed scale out servers to 192 core scale up servers LC and L line servers have been developed specifically for scale out Linux applications LC servers provide clear price/performance leadership with form factors and price points competitive with commodity Intel Unlike commodity Intel, Power Systems supports an open ecosystems via the OpenPOWER initiative which ensures continuous leading innovation For example: Intel per core performance generally stays flat each new generation while the number of cores per server increases driving up SW costs Power per core performance has demonstrated significant improvement with each successive generation. [NEXT SLIDE]
12. We know that IT system downtime is much more than an inconvenience or aggravation. When your systems are down or not performing optimally, people across your company can’t do their jobs effectively—if at all. As a result, revenue, profit, company reputation and customer loyalty can suffer. We know that you need one dependable, trusted provider to help manage and resolve problems regardless of the platform or vendor. In particular, you need open source and Linux infrastructure specialists. IBM has been committed to open technologies from the onset, bringing more than 16 years of experience supporting open source environments. Through our alliances with SUSE, Ubuntu and Red Hat, we demonstrate our commitment to the success of open technologies. We bring a virtually unparalleled expertise in supporting Linux across all IBM systems and OEM x86 platforms certified for Linux. Our well-established global support infrastructure is key. With that you can gain: In-depth skills and experience that encompass the many products and technologies that make up your IT environments A streamlined process that expedites problem determination and resolution A proven track record in delivering result-producing support services The ability to access your support provider as often as needed, around the clock and around the world With IBM, you can gain accountability. When you report a problem, you can feel confident that it will be handled professionally and expeditiously, and that our specialists will manage it diligently from initial report through resolution.
13. We know that IT system downtime is much more than an inconvenience or aggravation. When your systems are down or not performing optimally, people across your company can’t do their jobs effectively—if at all. As a result, revenue, profit, company reputation and customer loyalty can suffer. We know that you need one dependable, trusted provider to help manage and resolve problems regardless of the platform or vendor. In particular, you need open source and Linux infrastructure specialists. IBM has been committed to open technologies from the onset, bringing more than 16 years of experience supporting open source environments. Through our alliances with SUSE, Ubuntu and Red Hat, we demonstrate our commitment to the success of open technologies. We bring a virtually unparalleled expertise in supporting Linux across all IBM systems and OEM x86 platforms certified for Linux. Our well-established global support infrastructure is key. With that you can gain: In-depth skills and experience that encompass the many products and technologies that make up your IT environments A streamlined process that expedites problem determination and resolution A proven track record in delivering result-producing support services The ability to access your support provider as often as needed, around the clock and around the world With IBM, you can gain accountability. When you report a problem, you can feel confident that it will be handled professionally and expeditiously, and that our specialists will manage it diligently from initial report through resolution.
14. TALK TRACK POWER8’s leading IO and Memory capabilities power fast data access and movement across a wide range of BDA applications. POWER8 is the first microprocessor designed for Big Data and Analytics. When systems are designed for big data, there are a couple of key attributes that are important to create a balanced system design. First having the processing capability, second having the memory space, the workspace, and the third is having the bandwidth, the ability to move the information in and out of the system at the rapid speeds required. POWER8 delivers 4 times more threads per core vs. commodity infrastructure. We can easily support a growing number of users who need reports, or to perform ad hoc analytics. This is because the processor can run more concurrent queries in parallel faster, across multiple cores with more threads per core. We’re delivering up to 4 times more memory bandwidth. Increased memory bandwidth to access up to 1 TB of memory for data operations and enlarged cache in every processor. This delivers the levels of performance your teams need to make decisions in real time. We’re delivering faster IO and high speed caching to ingest, move and access large volumes of data so that analytics results are available faster. Moore’s law is no longer delivering the processor speed up needed to meet the demands of Big Data, the OpenPOWER foundation is driving innovations such as hardware accelerators that will deliver on a continuous improvement in price/performance leadership. Power Systems provide the capabilities needed to handle the varying analytics initiatives your business requires. Broad range of data and analytics – from operational to computational to business analytics, as well as cognitive solutions leveraging IBM Watson technology, Power Systems are optimized for performance and can scale to support demanding and growing workloads. These solutions help you capitalize on the currency of data by finding business insights faster and more efficiently. Supporting Data: threads per core vs. x86 (up to 1536 threads per system) memory bandwidth vs. x86 (up to 32TB of memory) more cache vs. x86 (>19MB cache per core) The POWER8 cache numbers are L1 - 96 KB per Core ==> L2 - 512KB per Core ==> L3 - 8 MB per core ==>(8192KB) L4 - 128 MB per Socket  128/12 = 10.7MB per core (10922KB per core) Total Cache per Core - 19.26MB (19722KB) The Intel side (Broadwell-EP) also has L1 and L2... L1 - 64 KB per Core ==> L2 - 256KB per Core ==> L3 - 55 MB per socket ==> 2.5MB per Core (2560KB per core) Total Cache per Core – 2.81MB (2880KB) Ratio is 19722/2880 = 6.85  round down to 6X
15. TALK TRACK - One good example of OpenPOWER innovation that could be exploited by GPU aware applications on Hadoop is the POWER8 NVLink support - In partnership with IBM’s OpenPOWER partner Nvidia, we released a POWER8 Linux server in Sept 2016 with 2.8X the IO throughput from CPU to GPU of what is possible on x86 - This removes the PCIe data pipe bottleneck to ensure full exploitation of the GPUs and is only available today on OpenPOWER servers Links: IBM S822LC for HPC with NVLink: http://www-03.ibm.com/systems/power/hardware/s822lc-hpc/ Related press: https://www.hpcwire.com/2016/09/08/ibm-debuts-power8-chip-nvlink-3-new-systems/ http://www.pcworld.com/article/3117718/ibms-new-power8-server-packs-in-nvidias-speedy-nvlink-interconnect.html
16. Note: This chart compares 16-c S822LC to the 36-c HP DL380. So the below System-level performance is more compelling when this is considered. S822LC system-level total through put is 20%+ higher than DL380. The price Performance shown is outcome of this higher performance and the lower HW street price of S822LC. On Trans/$ graph, note that the VM quantities show 20/24 VMs on Power and 16/20 VMs on x86 are intentionally different. As the goal is to show virtual machines that Produce similar per VM performance through put level. The slight raise in x86’s second bar is not significant and is due to the fact that x86 system is under allocated on first data point and not on second, where as p8 is already fully allocated on both data points shown. Memory: 256GB for S822LC and 256 GB for HP DL380 HBA: 2 x 16 gbps FCA for S822LC and HP Eth: 2port 10GbHDD: 2 x 1TB for S822LC & 2x 1TB for HP Price (match the configuration) BUT 2 x 300GB SAS 10k SFF for HP testcase Power S822LC delivers leadership performance vs Intel Xeon E5-2699 v3 2.7X more transactions/second per core Improve your revenue potential – Power S822LC delivers 25% more VMs in the same rack space as HP DL380 25% more virtual machines per server than HP DL380 76 more revenue generating virtual machines per rack than HP DL380
17. TALK TRACK - Spark workloads are ideal to gain advantage from the core capabilities of the POWER8 processor for multithreading and large memory - Streaming and SQL workload can be broken down into many parallel threads and exploit the 8 thread per core unique to POWER8 - Complex, memory intensive workloads like machine learning and graph, can benefit from the large memory bandwidth and cache - The result shown here are from an open source Spark Bench workload (contributed by IBM) which drives a set of representative workloads across machine learning, SQL and Graph on Spark. (https://github.com/SparkTC/spark-bench) - You can see that overall POWER8 provides a 2X per core advantage and 1.5X price performance advantage in these 7 node cluster results
18. TALK TRACK - To summarize, the combination of HDP and OpenPOWER offers the fastest route to exploitation through open community innovation - ODPi and OpenPOWER foundations ensure no risk of vendor lock-in and no other Hadoop + System combination can offer this pairing - Hortonworks and IBM both have deep industry experience and a proven history of outstanding dedication to client success and support - Ultimately, clients need fast time to insights and OpenPOWER delivers on performance with a reduced infrastructure cost
19. TALK TRACK - As mentioned earlier, the Hortonworks and IBM Power Systems partnership was announced at IBM Edge conference in Sept 2016 - Early adopter clients will gain access to technical preview builds in late 4Q2016 - General availability for HDP 2.5 for Linux on Power will follow in 1Q2017 - Future HDP releases will provide Linux on Power support at the same time as the Linux on Intel releases
20. TALK TRACK - If you are not already a member, join the Hortonworks community to access to valuable training resources and member insights - Get to know more about the value of Power Systems and OpenPOWER and how it can provide a unique, open and optimized alternative for your Big Data and Analytics workloads - To learn more about HDP and Power Systems, please join us for an upcoming webiniar or catch the replay.