Our major updates for April and May are the addition of Variational Autoencoders to the CCFRAUD example, the integration of Private Set Intersection for Vertical FL, and the introduction of Private Sandboxes supporting private AzureML workspaces.
- Enriched the CCFRAUD example to support Variational Autoencoders for faster Vertical FL (instructions to run the example here, additional information there).
- Introduced support for Private Set Intersection (preliminary step to a Vertical FL workflow) by leveraging the APSI library from MSR. More information here.
- Added a new type of sandbox (Private Sandboxes) that supports private AzureML workspaces (i.e. workspaces behind a VNet).
- Simplified the Kaggle credentials part of the Bank Marketing tutorial.
- Revamped our documentation for Vertical FL to make it more clear and accessible.
- Fixed a few issues in FL Experience (pytorch-version related bug) and Provisioning (uncaught validation issues with silos identities, deployment name conflicts for sandboxes, old AKS API's).
We are changing the release process of our FL Accelerator repository. We are moving from monthly releases to continuous releases. We will however keep updating this changelog once a month to highlight the most significant changes.
Our major updates for this month are the addition of resources to provision various flavors of FL sandboxes, a new example for Vertical FL, and better support for confidentiality (encryption at rest, communications in Vertical FL).
- Added support for data encryption at rest to the CCFRAUD example (instructions here).
- Introduced new Vertical FL example: Bank Marketing Campaign Prediction.
- Improved the communications in Vertical FL jobs (support for redis streams and encrypted communications, better logging to measure communications overhead).
- Released a new bicep script to deploy silos with AKS clusters using confidential computes and set up open orchestrator.
- Added support for including Kaggle credentials during workspace provisioning (useful for downloading Kaggle data to run our examples); see instructions included in our real-world examples documentation, or standalone instructions here.
- Added a "confidentiality keyvault" to our provisioning scripts to host a custom key for encryption at rest during preprocessing and training.
- Added some documentation and the associated bicep scripts to easily deploy sandboxes for exploring different variants of FL setups.
- Added instructions on how to leverage MLOps to restrict FL to peer-reviewed code.
- Updated our generic instructions to support encryption of the uploaded data using confidentiality keyvault.
- Added more CI/CD tests for the examples introduced last month.
- Added unit tests for communication and encryption components.
We are excited to announce the release of the February iteration of our FL Accelerator repository.
Some of the major updates we have made include the launch of a vertical federated learning feature, an FL pipeline that offers a native AML FL experience integrated with the factory engine, and benchmark results that reveal a comprehensive comparison between FL and non-FL experiments.
- Implemented Vertical Federated Learning and offered a tutorial to run MNIST or CCFRAUD examples.
- Introduced a scatter-gather pipeline that delivers a real AML FL native experience.
- Conducted a comprehensive comparison b/w FL and non-FL experiments and the benchmark report can be accessed here.
- Provided instructions and a script to facilitate the upload of local data to a silo storage.
- Incremental improvements:
- Enhanced the network security rules and minimized the workspace dependencies for provisioning resources.
To get started, go here!
If you find a bug or have a feature request, please open an issue on the GitHub repository.
We are excited to announce the release of the January iteration of our FL Accelerator repository.
Here below we list all the new features. The most significant changes are the introduction of a guide to help onboard people to FL on Azure ML, the implementation of Differential Privacy in all 3 industry-relevant examples, the support of third-party FL frameworks (NVFlare and Flower), and the support of distributed training.
- Implemented Differential Privacy in all 3 industry-relevant examples, and added a tutorial that explains the process.
- Introduced experimental support for third-party FL frameworks. The pneumonia example can now be run using NVFlare (see tutorial) or Flower (see tutorial).
- Implemented distributed training (in each silo individually) in all 3 industry-relevant examples for scalable training.
- Introduced support for multiple computes per silo (for instance: to use CPU's for pre-processing and GPU's for training).
- Introduced resources for Exploratory Data Analysis on the credit card fraud example.
- Incremental improvements:
- Made sure component scripts can run locally to facilitate authoring and debugging.
- Fixed a bug about data loading for the MNIST example.
- Added instructions on how to properly configure Confidential Computes so all of the resources can be properly utilized.
- Updated the bicep templates to also allow for GPU's provisioning (not restricted to CPU's anymore).
To get started, go here!
If you find a bug or have a feature request, please open an issue on the GitHub repository.
We are excited to announce the release of the November iteration of our FL Accelerator repository.
Here below we list all the new features. The most significant ones are the addition of 3 industry-relevant examples (credit card fraud detection, named entity recognition, pneumonia detection), and the introduction of a Troubleshooting Guide.
- Added 3 industry-relevant examples. The examples include 2 jobs. The first one (generic) downloads some public data, partitions them, and uploads them to the silos. The second job (example-specific) trains a model using FL. The 3 examples address the following topics:
- credit card fraud detection (finance example);
- named entity recognition (NLP example);
- pneumonia detection (medical imaging example).
- Fixed a bug causing the contents of the
config.jsonfile to take precedence over the CLI arguments (subscription id, resource group, workspace name) when submitting a job. Now, the CLI arguments takes precedence over theconfig.yamlfile, which itself takes precedence over theconfig.jsonfile.
- Introduced a new Troubleshooting Guide to help you troubleshoot common issues. We will keep adding to it as we become aware of more common issues.
- Improved the documentation home page by adding some pictures and introducing the industry-relevant examples.
- Revisited our CI/CD processes for better agility (enable concurrent jobs, accommodate token expiration issue).
We are excited to announce the release of the October iteration of our FL Accelerator repository.
Here are the new features.
- A new provisioning cookbook so you can pick and choose the setup that best suits your needs, based on the provided templates (orchestrator behind vnet, silos behind vnets, silos using an AKS cluster with confidential compute, silos using an existing storage account...).
- All bicep scripts can now be used as standalone to adapt to your specific infrastructure needs.
- Instructions on how to provision external silos.
- Tutorial on how to adapt literal and factory code to your scenarios.
- Introduced "one-click deployments" for all bicep templates.
- Using AzureML SDK subgraphs (preview) as a basis for the factory code sample, so that writing an FL experiment is easier and more flexible, and the experiment graph is shown better in the Azure ML UI.
- Added Subscription ID, Resource Group, and Workspace Name as CLI arguments for job submission.
- Enabled CI/CD on the repo to ensure nothing breaks.
We are excited to announce the release of the September iteration of our FL Accelerator repository.
Here are the new features.
- Cleaned up repository - removed redundant directories/files.
- Auto-provisioning scripts to create a complete sandbox using bicep templates (by default an open sandbox on 3 regions). An experimental script with VNets and private endpoints is also available.
- A quickstart to run an FL demo in minutes, and easily run FL jobs on your own data!
- Single location for all documentation.
- Easy navigation using Azure Storage Explorer to discover models of various iterations, here is a more intuitive path (<experiment-name>/<iteration-num>/<model-name>)
- Introduction of "factory" code to simplify the creation of FL experiments.
- Decoupled components for more readability.
- Soft validation to check the correct permissions of your assets.
- Metrics:
- The combined losses and model performances of several silos can now be seen at the pipeline level.