Spark and Spark History 2.6.0-2.3.2

New features

• Upgraded Spark and Spark History Server to 2.3.2
• Added DC/OS Spark CLI support for --jars
• Added CNI Support for Dispatcher, Driver, and Executors for Docker and Mesos containerizers
• Added CNI labels support for Mesos containerizer
• Added package configuration for CNI:
  • virtual_network_enabled
  • virtual_network_name
  • virtual_network_plugin_labels
• Spark Dispatcher by default launches Spark Drivers and Executors in the same virtual network it was launched in itself

Updates

• SPARK_HOME environment variable defaults to /opt/spark in Dockerfile and executable scripts
• Switched to Spark's own StatsD Sink instead of 3rd-party dependency
• Updated dcos-commons bootstrap version to 0.55.2

Bug fixes

• Fixed bug for Dispatcher restarting duplicate Spark drivers after agent restart in --supervised mode
• Fixed bug for CLI incorrect --jars parsing resulting in submit failure

Limitations

• Configuration of network plugin labels from DC/OS UI supported only in JSON editing mode
• Network plugin labels are not supported by Docker containerizer
• Currently, DC/OS AdminRouter doesn't support virtual networks so DC/OS Spark endpoints will not be accessible from CLI if Spark installed with virtual network enabled. Spark jobs need to be submitted from a routable network in this case