Today we're announcing updates to Apache Spark, Apache Kafka, Machine Learning Services, and Azure Data Lake Storage Gen2, and enhancements to Enterprise Security Package. These new capabilities will continue to drive savings for many customers. In addition, Microsoft is deepening its commitment to the Apache Hadoop ecosystem and has extended its partnership with Hortonworks to bring the best of Apache Hadoop and open-source big data analytics to the cloud.
We are thrilled to introduce support for Azure Data Lake (ADL) Python and R extensions within Visual Studio Code (VSCode). This means you can easily add Python or R scripts as custom code extensions in U-SQL scripts, and submit such scripts directly to ADL with one click. For data scientists who value the productivity of Python and R, ADL Tools for VSCode offers a fast and powerful code editing solution. VSCode makes it simple to get started and provides easy integration with U-SQL for data extract, data processing, and data output.
Azure Data Lake Tools for Visual Studio now provides the ability to export all or part of a database associated with an ADL Analytics (ADLA) account into a database on a local machine. This capability is very useful when creating local environment for development or debugging.
The Azure Data Lake Analytics service can now help you easily organize, manage, and gain insights from Data Lake Analytics jobs that are running as part of pipelines or on a recurring basis. New capabilities enable you to:
Quickly identify jobs in pipelines that might have failed or taken longer than expected.
Get the aggregated statistics (job counts, successful and failed AU hours, and so on) for a pipeline or a recurring instance so you can better understand the resource consumption and cost trends.
Support for Azure Data Lake Store (ADLS) is now available in Azure Analysis Services and in SQL Server Data Tools (SSDT). Now you can augment your big data analytics workloads with rich interactive analysis for selected data subsets at the speed of thought! Business users can consume Azure Analysis Services models in Microsoft Power BI, Microsoft Office Excel, and Microsoft SQL Server Reporting Services. Azure Data Lake Analytics (ADLA) can be used to run U-SQL batch jobs directly against the source data, such as to generate targeted output files that Azure Analysis Services can import with less overhead.
The July Update for Azure Data Factory has been posted. Included, but not limited to, Preview for Data Management Gateway high availability and scalability Skipping or logging incompatible rows during copy for fault tolerance Service principal authentication support for Azure Data Lake Analytics.
Azure Data Lake Analytics now supports new policies that help you control the compute resources allocated to your most business-critical and experimental/ad-hoc jobs that are running side-by-side in the same account. The policies can be customized to meet your unique business and cost control requirements.