Statistical Inference using Stochastic Gradient Descent: Volumes 1 and 2

Volume 1: The authors present a novel inference framework for convex empirical risk minimization, using approximate stochastic Newton steps. The proposed algorithm is based on the notion of finite differences and allows the approximation of a Hessian-vector product from first-order information. In theory, the authors' method efficiently computes the statistical error covariance in M-estimation, both for unregularized convex learning problems and high-dimensional LASSO regression, without using exact second order information, or resampling the entire data set. In practice, the authors demonstrate the effectiveness of their framework on large-scale machine learning problems, that go even beyond convexity: as a highlight, the authors' work can be used to detect certain adversarial attacks on neural networks. Volume 2: The authors present a novel method for frequentist statistical inference in M-estimation problems, based on stochastic gradient descent (SGD) with a fixed step size: the authors demonstrate that the average of such SGD sequences can be used for statistical inference, after proper scaling. An intuitive analysis using the Ornstein-Uhlenbeck process suggests that such averages are asymptotically normal. From a practical perspective, the authors' SGD-based inference procedure is a first order method, and is well-suited for large scale problems. To show its merits, the authors apply it to both synthetic and real datasets, and demonstrate that its accuracy is comparable to classical statistical methods, while requiring potentially far less computation.

• Record URL:
  https://ctr.utexas.edu/wp-content/uploads/144.pdf
• 
• Summary URL:
  https://library.ctr.utexas.edu/Presto/catalogid=36851
• Record URL:
  https://rosap.ntl.bts.gov/view/dot/66276
• Supplemental Notes:
  • “This document was sponsored by the U.S. Department of Transportation, University Transportation Centers Program.”
• Corporate Authors:

  University of Texas, Austin

  Data-Supported Transportation Operations & Planning Center (D-STOP)
  3925 West Braker Lane, 4th Floor
  Austin, TX  United States  78759

  University of Texas, Austin

  Wireless Networking and Communications Group
  ,    

  University of Texas, Austin

  Center for Transportation Research
  3925 W. Braker Lane, 4th Floor
  Austin, TX  United States  78759

  Department of Transportation

  University Transportation Centers Program, 1200 New Jersey Avenue, SE
  Washington, DC  United States  20590

  Office of the Assistant Secretary for Research and Technology

  University Transportation Centers Program
  Department of Transportation
  Washington, DC  United States  20590
• Authors:
  • Juri, Natalia Ruiz
  • Boyles, Stephen D
  • Zhu, Tengkuo
  • Perrine, Kenneth
  • Chen, Amber
  • Li, Yun
  • Caramanis, Constantine
• Publication Date: 2018-10

Language

• English

Media Info

• Media Type: Digital/other
• Features: Figures; References;
• Pagination: 99p

Subject/Index Terms

• TRT Terms: Computer security; Risk management; Statistical inference; Stochastic processes; Wireless communication systems
• Subject Areas: Operations and Traffic Management; Planning and Forecasting; Transportation (General);

Filing Info

• Accession Number: 01700374
• Record Type: Publication
• Report/Paper Numbers: D-STOP/2018/144, Report 144
• Contract Numbers: DTRT13-G-UTC58
• Files: UTC, NTL, TRIS, ATRI, USDOT
• Created Date: Mar 29 2019 10:20AM