We are excited to announce the general availability of private endpoint in HDInsight clusters deployed in a virtual network. This feature enables enterprises to better isolate access to their HDInsight clusters from the public internet and enhance their security at the networking layer.
Bigger organizations can now securely deploy the Azure HDInsight service. You can create an HDInsight cluster with Enterprise Security Package (ESP) that's joined to an Azure Active Directory Domain Services (Azure AD DS) managed domain. You can then configure a list of employees from the enterprise who can authenticate through Azure AD DS to log on to the HDInsight cluster.
This GA release includes a greatly simplified cluster setup experience that takes advantage of integration between Azure AD DS and the HDInsight service.
Today, we are pleased to reveal the preview of HDInsight IO Cache, a new transparent data caching feature of Azure HDInsight that provides customers with up to a 9x performance improvement for Apache Spark jobs. We know from our customers that when it comes to analytics cost efficiency of managed cloud-based Apache Hadoop and Spark services is one of their major attractors. HDInsight IO Cache allows us to improve this key value proposition even further by improving performance without a corresponding increase in costs.
The HDInsight team is excited to announce the general availability of Enterprise Security Package (ESP) for Apache Spark, Apache Hadoop and Interactive Query clusters in HDInsight 3.6. When enterprise customers share clusters between multiple employees, Hadoop admins must ensure those employees have the right set of accesses and permissions to perform big data operations. In enterprises, multi-user access with granular authorization using the same identities in the enterprise is a complex and lengthy process. Enabling ESP with the new experience provides authentication and authorization for these clusters in a more streamlined and secure manner.
We are expanding ESP to HDInsight 3.6 Apache Kafka and Apache HBase clusters. Apache Kafka and HBase customers can now use domain accounts and Apache Ranger for Authentication and Authorization. Enabling ESP means Apache Kafka and HBase Ranger plugins will be available out of the box. To learn more about secure Kafka clusters, see our documentation.
Earlier today, Microsoft Corporation announced its continuing support and commitment to enterprises seeking to use Hadoop for open source big data analytics in the cloud. Leading off the series of major upgrades to the Azure HDInsight service is the preview release of Hadoop 3.0, the transformational update to the Hadoop stack that enterprises have been waiting for since earlier this year. In addition, enterprises with strict security and compliance requirements will be able to secure their Azure HDInsight clusters using Enterprise Security Package. And there is something in this release for everybody! Spark developers will particularly like the series of innovations from Microsoft that will now allow them to quickly identify and resolve performance bottlenecks in their code.