Nonlinear Programming, 3rd Edition, 2016

Convex Optimization Theory, 2009

Convex Optimization Algorithms, 2015

Constrained Optimization and Lagrange Multiplier Methods, 1982

This series of complementary textbooks cover all aspects of continuous optimization, and its connections with discrete optimization via duality. The two convex optimization books deal primarily with convex, possibly nondifferentiable, problems and rely on convex analysis. By contrast the nonlinear programming book focuses primarily on analytical and computational methods for possibly nonconvex differentiable problems. It relies primarily on calculus and variational analysis, yet it still contains a detailed presentation of duality theory and its uses for both convex and nonconvex problems.

The books share notation, style, and prerequisites, and are aimed at students, researchers, and practitioners, roughly at the first year graduate level. They rely on rigorous analysis, but they also aim at an intuitive exposition that makes use of geometric visualization. They are supported by substantial internet-accessible material, including exercises with solutions, lecture slides, and course content from MIT Open Courseware. Three other complementary and more specialized books of the author can also be freely downloaded: the 1998 Network Optimization book, the 1989 Parallel and Distributed Computation book (coauthored with J. Tsitsiklis), and the 1982 Constrained Optimization and Lagrange Multiplier Methods book.

Instructors that adopt any one book as a main textbook for a course may contact the publishing company Athena Scientific for a free adoption copy.

This book provides an up-to-date, comprehensive, and rigorous account of nonlinear programming at the first year graduate student level. It covers descent algorithms for unconstrained and constrained optimization, Lagrange multiplier theory, interior point and augmented Lagrangian methods for linear and nonlinear programs, duality theory, and major aspects of large-scale optimization.

The third edition of the book is a thoroughly rewritten version of the 1999 second edition. New material was included, some of the old material was discarded, and a large portion of the remainder was reorganized or revised. The new material covers a variety of topics, such as first order methods, proximal algorithms, alternating direction methods of multipliers, and conic programming; also large-scale optimization topics of much current interest, such as incremental methods, and distributed asynchronous computation, and their applications in machine learning, signal processing, neural network training, and big data applications.

Another objective of the third edition to integrate its contents with those of the other books, together with internet-accessible material, so that they complement each other and form a unified whole. To this end, bridges to the other books have been provided, with extensive references to generalizations, discussions, and elaborations of the analysis.

Click here to download the Table of Contents and the Preface.

Click here to download the Newton Method code referenced in the last appendix of the book.

This book aims at an insightful, concise, and rigorous treatment of the basic theory of convex sets and functions in finite dimensions, and the analytical/geometrical foundations of convex optimization and duality theory. It provides much of the mathematical background needed for the in-depth reading of the Nonlinear Programming and Convex Optimization Algorithms books.

Convexity theory is first developed in a simple accessible manner, using easily visualized proofs. Then the focus shifts to a transparent geometrical line of analysis to develop the fundamental duality between descriptions of convex functions in terms of points, and in terms of hyperplanes. Finally, convexity theory and abstract duality are applied to problems of constrained optimization, Fenchel and conic duality, and game theory to develop the sharpest possible duality results within a highly visual geometric framework.

The book is aimed at students, researchers, and practitioners, roughly at the first year graduate level. It is similar in style to the author's 2015 Convex Optimization Algorithms book, but can be read independently. The latter book focuses on algorithmic issues, while the 2009 Convex Optimization Theory book focuses on convexity theory and optimization duality. The two books share notation, and together cover the entire finite-dimensional convex optimization methodology.

The paper Stochastic Optimization Problems with Nondifferentiable Cost Functionals, JOTA, 1973, on differentiation of the expected value of nondifferentiable convex functions.

Mathematical statements of Danskin's Theorem, on differentiation of the maximum of an infinite collection of convex functions.

This book provides a comprehensive and accessible presentation of algorithms for solving convex optimization problems. It relies on rigorous mathematical analysis, but also aims at an intuitive exposition that makes use of visualization where possible. This is facilitated by the extensive use of analytical and algorithmic concepts of duality, which by nature lend themselves to geometrical interpretation. The book places particular emphasis on modern developments, and their widespread applications in fields such as large-scale resource allocation problems, signal processing, and machine learning.

The book is aimed at students, researchers, and practitioners, roughly at the first year graduate level. It is similar in style to the author's 2009 Convex Optimization Theory book, but can be read independently. The latter book focuses on convexity theory and optimization duality, while the 2015 Convex Optimization Algorithms book focuses on algorithmic issues. The two books share notation, and together cover the entire finite-dimensional convex optimization methodology. To facilitate readability, the statements of definitions and results of the "theory book" are reproduced without proofs in Appendix B.

The print version of the book is available from the publishing company Athena Scientific, or from Amazon.com. It is also available as an Ebook from Google Play.

Click here to visit the book's web site at Athena Scientific for exercises with solutions, slides, and other instructional material, or to order the book directly from the publisher.

Click here to download Preface and Table of Contents, and Chapter 1

Constrained Optimization and Lagrange Multiplier Methods

This research monograph, first published in 1982 by Academic Press, and republished by Athena Scientific in 1996, is a comprehensive treatment of some of the most widely used constrained optimization methods, including the augmented Lagrangian/multiplier and sequential quadratic programming methods.

"This is an excellent reference book. The author has done a great job in at least three directions. First, he expertly, systematically and with ever-present authority guides the reader through complicated areas of numerical optimization. This is achieved by carefully explaining and illustrating (by figures, if necessary) the underlying principles and theory. Second, he provides extensive guidance on the merits of various types of methods. This is extremely useful to practitioners. Finally, this is truly a state of the art book on numerical optimization."
S. Zlobec, McGill University, in SIAM Review

It can be purchased from Athena Scientific or as EBOOK from Google Play. It can also be freely downloaded in scanned form

Constrained Optimization and Lagrange Multiplier Methods

1. Introduction

   1. General Remarks
   2. Notation and Mathematical Background
   3. Unconstrained Optimization
   4. Constrained Minimization
   5. Algorithms for Minimization Subject to Simple Constraints
   6. Notes and Sources
   
   

2. The Method of Multipliers for Equality Constrained Problems

   1. The Quadratic Penalty Function Method
   2. The Original Method of Multipliers
   3. Duality Framework for the Method of Multipliers
   4. Multiplier Methods with Partial Elimination of Constraints
   5. Asymptotically Exact Minimization in the Method of Multipliers
   6. Primal-Dual Methods Not Utilizing a Penalty Function
   7. Label Correcting Methods
   8. Notes and Sources
   
   

3. The Method of Multipliers for Inequality Constrained and Nondifferentiable Optimization Problems

   1. One-Sided Inequality Constraints
   2. Two-Sided Inequality Constraints
   3. Approximation Procedures for Nondifferentiable and Ill-Conditioned Optimization Problems
   4. Notes and Sources
   
   

4. Exact Penalty Methods and Lagrangian Methods

   1. Nondifferentiable Exact Penalty Functions
   2. Linearization Algorithms Based on Nondifferentiable Exact Penalty Functions
   3. Differentiable Exact Penalty Functions
   4. Lagrangian Methods - Local Convergence
   5. Lagrangian Methods - Global Convergence
   6. Notes and Sources
   
   

5. Nonquadratic Penalty Functions - Convex Programming

   1. Classes of Penalty Functions and Corresponding Methods of Multipliers
   2. Convex Programming and Duality
   3. Convergence Analysis of Multiplier Methods
   4. Rate of Convergence Analysis
   5. Conditions for Penalty Methods to be Exact
   6. Large Scale Integer Programming Problems and the Exponential Method of Multipliers
   7. Notes and Sources
   
   

6. References

7. Index