Communication-efficient Distributed Inference: Distributions, Approximation, and Improvement

Abstract

In modern data science, it is common that large-scale data are stored and processed parallelly across a great number of locations. For reasons including confidentiality concerns, only limited data information from each parallel center is eligible to be transferred. To solve these problems more efficiently, a group of communication-efficient methods are being actively developed. The first part of our investigation is the distributions of the distributed M-estimators that require a one-step update, combining data information collected from all parallel centers. We reveal that the number of centers plays a critical role. When it is not small compared with the total sample size, a non-negligible impact occurs to the limiting distributions, which turn out to be mixtures involving products of normal random variables. Based on our analysis, we propose a multiplier-bootstrap method for approximating the distributions of these one-step updated estimators. Our second contribution is that we propose two communication-efficient Newton-type algorithms, combining the M-estimator and the gradient collected from each data center. They are created by constructing two Fisher information estimators globally with those communication-efficient statistics. Enjoying a higher rate of convergence, this framework improves upon existing Newton-like methods. Moreover, we present two bias-adjusted one-step distributed estimators. When the square of the center-wise sample size is of a greater magnitude than the total number of centers, they are as efficient as the global M-estimator asymptotically. The advantages of our methods are illustrated by extensive theoretical and empirical evidences.