Publications

Disclaimer: This material is presented to ensure the timely dissemination of scholarly works. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms invoked by each author's copyright.

* denotes equal contribution and joint lead authorship

2023

1. IEEE TAI

   

   A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design

   Mahdi Morafah, Weijia Wang, and Bill Lin

   IEEE Transactions on Artificial Intelligence 2023

   Abs arXiv Bib Code

   Federated Learning (FL) has been an area of active research in recent years. There have been numerous studies in FL to make it more successful in the presence of data heterogeneity. However, despite the existence of many publications, the state of progress in the field is unknown. Many of the works use inconsistent experimental settings and there are no comprehensive studies on the effect of FL-specific experimental variables on the results and practical insights for a more comparable and consistent FL experimental setup. Furthermore, the existence of several benchmarks and confounding variables has further complicated the issue of inconsistency and ambiguity. In this work, we present the first comprehensive study on the effect of FL-specific experimental variables in relation to each other and performance results, bringing several insights and recommendations for designing a meaningful and well-incentivized FL experimental setup. We further aid the community by releasing FedZoo-Bench, an open-source library based on PyTorch with pre-implementation of 22 state-of-the-art methods, and a broad set of standardized and customizable features. We also provide a comprehensive comparison of several state-of-the-art (SOTA) methods to better understand the current state of the field and existing limitations.

   @article{morafah2023practical,
     title = {A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Design},
     author = {Morafah, Mahdi and Wang, Weijia and Lin, Bill},
     journal = {IEEE Transactions on Artificial Intelligence},
     year = {2023},
     abbr = {IEEE TAI},
     organization = {IEEE},
     arxiv = {2307.15245},
     bibtex_show = {true},
     code = {https://github.com/MMorafah/FedZoo-Bench},
     selected = {true}
   }

2022

1. AAAI

   

   Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces

   Saeed Vahidian*, Mahdi Morafah*, Weijia Wang, Vyacheslav Kungurtsev, Chen Chen, Mubarak Shah, and Bill Lin

   arXiv preprint arXiv:2209.10526 2022

   Abs arXiv Bib Code

   Clustered federated learning (FL) has been shown to produce promising results by grouping clients into clusters. This is especially effective in scenarios where separate groups of clients have significant differences in the distributions of their local data. Existing clustered FL algorithms are essentially trying to group together clients with similar distributions so that clients in the same cluster can leverage each other’s data to better perform federated learning. However, prior clustered FL algorithms attempt to learn these distribution similarities indirectly during training, which can be quite time consuming as many rounds of federated learning may be required until the formation of clusters is stabilized. In this paper, we propose a new approach to federated learning that directly aims to efficiently identify distribution similarities among clients by analyzing the principal angles between the client data subspaces. Each client applies a truncated singular value decomposition (SVD) step on its local data in a single-shot manner to derive a small set of principal vectors, which provides a signature that succinctly captures the main characteristics of the underlying distribution. This small set of principal vectors is provided to the server so that the server can directly identify distribution similarities among the clients to form clusters. This is achieved by comparing the similarities of the principal angles between the client data subspaces spanned by those principal vectors. The approach provides a simple, yet effective clustered FL framework that addresses a broad range of data heterogeneity issues beyond simpler forms of Non-IIDness like label skews. Our clustered FL approach also enables convergence guarantees for non-convex objectives.

   @article{vahidian2022efficient,
     title = {Efficient Distribution Similarity Identification in Clustered Federated Learning via Principal Angles Between Client Data Subspaces},
     author = {Vahidian*, Saeed and Morafah, Mahdi and Wang, Weijia and Kungurtsev, Vyacheslav and Chen, Chen and Shah, Mubarak and Lin, Bill},
     journal = {arXiv preprint arXiv:2209.10526},
     year = {2022},
     abbr = {AAAI},
     arxiv = {2209.10526},
     bibtex_show = {true},
     code = {https://github.com/MMorafah/PACFL},
     selected = {true},
     equal = {true}
   }

2. IEEE OJCS

   

   FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution

   Mahdi Morafah*, Saeed Vahidian*, Weijia Wang*, and Bill Lin

   arXiv preprint arXiv:2208.09754 2022

   Abs arXiv Bib Code

   Classical federated learning approaches yield significant performance degradation in the presence of Non-IID data distributions of participants. When the distribution of each local dataset is highly different from the global one, the local objective of each client will be inconsistent with the global optima which incur a drift in the local updates. This phenomenon highly impacts the performance of clients. This is while the primary incentive for clients to participate in federated learning is to obtain better personalized models. To address the above-mentioned issue, we present a new algorithm, FLIS, which groups the clients population in clusters with jointly trainable data distributions by leveraging the inference similarity of clients’ models. This framework captures settings where different groups of users have their own objectives (learning tasks) but by aggregating their data with others in the same cluster (same learning task) to perform more efficient and personalized federated learning. We present experimental results to demonstrate the benefits of FLIS over the state-of-the-art benchmarks on CIFAR-100/10, SVHN, and FMNIST datasets.

   @article{morafah2022flis,
     title = {FLIS: Clustered Federated Learning via Inference Similarity for Non-IID Data Distribution},
     author = {Morafah, Mahdi and Vahidian*, Saeed and Wang*, Weijia and Lin, Bill},
     journal = {arXiv preprint arXiv:2208.09754},
     year = {2022},
     abbr = {IEEE OJCS},
     arxiv = {2208.09754},
     bibtex_show = {true},
     code = {https://github.com/MMorafah/FLIS},
     selected = {true},
     equal = {true}
   }

3. NeurIPS

   

   Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks

   Mahdi Morafah*, Saeed Vahidian*, Chen Chen, Mubarak Shah, and Bill Lin

   arXiv preprint arXiv:2209.15595 2022

   Abs arXiv Bib Code

   Though successful, federated learning presents new challenges for machine learning, especially when the issue of data heterogeneity, also known as Non-IID data, arises. To cope with the statistical heterogeneity, previous works incorporated a proximal term in local optimization or modified the model aggregation scheme at the server side or advocated clustered federated learning approaches where the central server groups agent population into clusters with jointly trainable data distributions to take the advantage of a certain level of personalization. While effective, they lack a deep elaboration on what kind of data heterogeneity and how the data heterogeneity impacts the accuracy performance of the participating clients. In contrast to many of the prior federated learning approaches, we demonstrate not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive: (1) Dissimilar labels of clients (label skew) are not necessarily considered data heterogeneity, and (2) the principal angle between the agents’ data subspaces spanned by their corresponding principal vectors of data is a better estimate of the data heterogeneity.

   @article{morafah2022rethinking,
     title = {Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks},
     author = {Morafah, Mahdi and Vahidian*, Saeed and Chen, Chen and Shah, Mubarak and Lin, Bill},
     journal = {arXiv preprint arXiv:2209.15595},
     year = {2022},
     abbr = {NeurIPS},
     arxiv = {2209.15595},
     bibtex_show = {true},
     code = {https://github.com/MMorafah/FL-SC-NIID},
     selected = {true},
     equal = {true}
   }

2021

1. ICDCSW Conference Award

   

   Personalized federated learning by structured and unstructured pruning under data heterogeneity

   Saeed Vahidian*, Mahdi Morafah*, and Bill Lin

   In 2021 IEEE 41st International Conference on Distributed Computing Systems Workshops (ICDCSW) 2021

   Abs arXiv Bib Code

   The traditional approach in FL tries to learn a single global model collaboratively with the help of many clients under the orchestration of a central server. However, learning a single global model might not work well for all clients participating in the FL under data heterogeneity. Therefore, the personalization of the global model becomes crucial in handling the challenges that arise with statistical heterogeneity and the non-IID distribution of data. Unlike prior works, in this work we propose a new approach for obtaining a personalized model from a client-level objective. This further motivates all clients to participate in federation even under statistical het- erogeneity in order to improve their performance, instead of merely being a source of data and model training for the central server. To realize this personalization, we leverage finding a small subnetwork for each client by applying hybrid pruning (combination of structured and unstructured pruning), and unstructured pruning. Through a range of experiments on different benchmarks, we observed that the clients with similar data (labels) share similar personal parameters. By finding a subnetwork for each client rather than taking the average over all parameters of all clients for the entire federation as in traditional FL, we efficiently calculate the averaging on the remaining parameters of each subnetwork of each client. We call this novel parameter averaging as Sub-FedAvg. Furthermore, in our proposed approach, the clients are not required to have knowledge of any underlying data distributions or label similarities among the rest of clients. The non-IID nature of each client’s local data provides distinguishing subnetworks without sharing any data. We evaluate our method on federated image classification with real world datasets. Our method outperforms existing state-of-the-art.

   @inproceedings{vahidian2021personalized,
     title = {Personalized federated learning by structured and unstructured pruning under data heterogeneity},
     author = {Vahidian*, Saeed and Morafah, Mahdi and Lin, Bill},
     booktitle = {2021 IEEE 41st International Conference on Distributed Computing Systems Workshops (ICDCSW)},
     pages = {27--34},
     year = {2021},
     organization = {IEEE},
     bibtex_show = {true},
     abbr = {ICDCSW},
     code = {https://github.com/MMorafah/Sub-FedAvg},
     arxiv = {2105.00562},
     award = {Conference Award},
     equal = {true},
     selected = {true}
   }

2. IEEE TCNS

   

   Decentralized Asynchronous Non-convex Stochastic Optimization on Directed Graphs

   Vyacheslav Kungurtsev, Mahdi Morafah, Tara Javidi, and Gesualdo Scutari

   arXiv preprint arXiv:2110.10406 2021

   Abs arXiv

   Distributed Optimization is an increasingly important subject area with the rise of multi-agent control and optimization. We consider a decentralized stochastic optimization problem where the agents on a graph aim to asynchronously optimize a collective (additive) objective function consisting of agents’ individual (possibly non-convex) local objective functions. Each agent only has access to a noisy estimate of the gradient of its own function (one component of the sum of objective functions). We proposed an asynchronous distributed algorithm for such a class of problems. The algorithm combines stochastic gradients with tracking in an asynchronous push-sum framework and obtain the standard sublinear convergence rate for general non-convex functions, matching the rate of centralized stochastic gradient descent SGD. Our experiments on a non-convex image classification task using convolutional neural network validate the convergence of our proposed algorithm across different number of nodes and graph connectivity percentages.