Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

A Hybrid Differential Dynamic Programming Algorithm for Constrained Optimal Control Problems. Part 1: Theory

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

A new algorithm is presented to solve constrained nonlinear optimal control problems, with an emphasis on highly nonlinear dynamical systems. The algorithm, called HDDP, is a hybrid variant of differential dynamic programming, a proven second-order technique that relies on Bellman’s Principle of Optimality and successive minimization of quadratic approximations. The new hybrid method incorporates nonlinear mathematical programming techniques to increase efficiency: quadratic programming subproblems are solved via trust region and range-space active set methods, an augmented Lagrangian cost function is utilized, and a multiphase structure is implemented. In addition, the algorithm decouples the optimization from the dynamics using first- and second-order state transition matrices. A comprehensive theoretical description of the algorithm is provided in this first part of the two paper series. Practical implementation and numerical evaluation of the algorithm is presented in Part 2.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8

Similar content being viewed by others

Notes

  1. Note that the original continuous optimal control problems can be solved also via indirect methods and optimal control theory through a multi-point boundary value problem formulation.

  2. Since δx k , δw, and δλ are unknown for a particular stage, the control update needs to be a function of these quantities.

  3. Note that \(\widetilde {J}_{uu,k}\) is known since a trust region subproblem was solved before to estimate the active constraints.

  4. It is easy to check that P satisfies the scaled projection identity \(P \widetilde{J}^{-1}_{uu,k}P = P\).

  5. No terms in δx k are present in the constant term ER since δx k is zero on the reference trajectory.

  6. For the last phase (i=M), when the final constraints are dependent on w +=w 1 and x +=x 1,1 (like in periodic constraints), all the last terms of right hand side of (54a)–(54j) are not known (the sensitivites \(\widehat{J}_{w+},\widehat{J}_{w+w+},\ldots\); the feedback terms A w+,B w+,…). We suggest that their values could be taken from the last iteration.

  7. When stage constraints are present, only the reduced Hessians should be positive definite.

  8. If the filtering method presented in Part 2 of the paper series is included, then the filtering condition should be satisfied in this step as well.

References

  1. Barclay, A., Gil, P.E., Rosen, J.B.: In: SQP Methods and Their Application to Numerical Optimal Control. International Series of Numerical Mathematics, vol. 124, pp. 207–222 (1998)

    Google Scholar 

  2. Powell, M.J.D.: A Method for Nonlinear Constraints in Minimization Problems. Academic Press, London and New York (1969). R. Fletcher (ed.), optimization edition

    Google Scholar 

  3. Hestenes, M.R.: Multiplier and gradient methods. J. Optim. Theory Appl. 4, 303–320 (1969)

    Article  MathSciNet  MATH  Google Scholar 

  4. Conn, A.R., Gould, G.I.M., Toint, P.L.: Lancelot: A Fortran Package for Large-Scale Nonlinear Optimization (Release A). Springer, Berlin (1992)

    MATH  Google Scholar 

  5. Murtagh, B.A., Saunders, M.A.: A projected Lagrangian algorithm and its implementation for sparse non-linear constraints. In: Algorithms for Constrained Minimization of Smooth Nonlinear Functions. Mathematical Programming Studies, vol. 16, pp. 84–117 (1982)

    Chapter  Google Scholar 

  6. Wilson, R.B.: A simplicial method for convex programming. PhD thesis. Harvard University (1963)

  7. Han, S.P.: A globally convergent method for nonlinear programming. J. Optim. Theory Appl. 22(3), 297–309 (1977)

    Article  MATH  Google Scholar 

  8. Powell, M.J.D.: The convergence of variable metric methods for nonlinearly constrained optimization calculations. In: Nonlinear Programming, vol. 3. Academic Press, San Diego (1978)

    Google Scholar 

  9. Gill, P.E., Murray, W., Saunders, M.A.: SNOPT: an SQP Algorithm for Large-Scale Constrained Optimization. SIAM J. Optim. 12(4), 979–1006 (2002)

    Article  MathSciNet  MATH  Google Scholar 

  10. Betts, J.T., Frank, P.D.: A sparse nonlinear optimization algorithm. J. Optim. Theory Appl. 82(3), 519–541 (1994)

    Article  MathSciNet  MATH  Google Scholar 

  11. Wachter, A., Biegler, L.T.: On the implementation of a primal-dual interior point filter line search algorithm for large-scale nonlinear programming, mathematical programming. Math. Program. 106(1), 25–57 (2006)

    Article  MathSciNet  Google Scholar 

  12. Nikolayzik, T., Buskens, C.: WORHP (We Optimize Really Huge Problems). In: 4th International Conference on Astrodynamics Tools and Techniques, Madrid, Spain, May (2010)

    Google Scholar 

  13. Gill, P.E., Murray, W., Saunders, M.A., Wright, M.H.: User’s guide for SOL/NPSOL: a Fortran package for nonlinear programming. Report sol 83-12. Department of Operations Research, Stanford University, California (1983)

  14. Kraft, D.: A software package for sequential quadratic programming. Technical report DFVLR-FB 88-28, Institut für Dynamik der Flugsysteme, Koln, Germany (July 1988)

  15. Vanderbei, R.J.: LOQO: an interior point code for quadratic programming. Optim. Methods Softw. 12, 451–484 (1999)

    Article  MathSciNet  Google Scholar 

  16. Byrd, R.H., Nocedal, J., Waltz, R.A.: KNITRO: an integrated package for nonlinear optimization. In: Large Scale Nonlinear Optimization, pp. 35–59. Springer, Berlin (2006)

    Chapter  Google Scholar 

  17. Powell, M.J.D.: Extensions to subroutine VF02. In: Drenick, R.F., Kozin, F. (eds.) System Modeling and Optimization. Lecture Notes in Control and Information Sciences, vol. 38, pp. 529–538. Springer, Berlin (1982)

    Chapter  Google Scholar 

  18. Harwell subroutine library. http://www.hsl.rl.ac.uk/

  19. Kraft, D.: Algorithm 733: Tomp–Fortran modules for optimal control calculations. ACM Trans. Math. Softw. 20(3), 262–281 (1994)

    Article  MATH  Google Scholar 

  20. Ross, I.M.: User’s manual for DIDO (ver. pr. 13): a Matlab application package for solving optimal control problems. Technical report 04-01.0, Naval Postgraduate School, Monterey, CA, February (2004)

  21. Rao, A.V., Benson, D.A., Darby, C., Patterson, M.A., Francolin, C., Sanders, I., Huntington, G.T.: Algorithm 902: GPOPS, a Matlab software for solving multiple-phase optimal control problems using the Gauss pseudospectral method. ACM Trans. Math. Softw. 37(2), 1–39 (2010)

    Article  Google Scholar 

  22. Sims, J.A., Finlayson, P., Rinderle, E., Vavrina, M., Kowalkowski, T.: Implementation of a low-thrust trajectory optimization algorithm for preliminary design. In: AAS/AIAA Astrodynamics Specialist Conference and Exhibit, Keystone, CO, August (2006). No. AIAA-2006-674

    Google Scholar 

  23. Franke, R.: Omuses tool for the optimization of multistage systems and HQP a solver for sparse nonlinear optimization, version 1.5. Technical report, Technical University of Ilmenau (1998)

  24. Gill, P.E., Jay, L.O., Leonard, M.W., Petzold, L.R., Sharma, V.: An SQP method for the optimal control of large-scale dynamical systems. J. Comput. Appl. Math. 120(1), 197–213 (2000)

    Article  MathSciNet  MATH  Google Scholar 

  25. Blaszczyk, J., Karbowski, A., Malinowski, K.: Object library of algorithms for dynamic optimization problems: benchmarking SQP and nonlinear interior point methods. Int. J. Appl. Math. Comput. Sci. 17(4), 515–537 (2007)

    Article  MathSciNet  MATH  Google Scholar 

  26. Lantoine, G.: A methodology for robust optimization of Low-Thrust trajectories in multibody environments. PhD thesis. School of Aerospace Engineering, Georgia Institute of Technology, Georgia (2010)

  27. Bullock, T.E.: Computation of optimal controls by a method based on second variations. PhD thesis. Department of Aeronautics and Astronautics, Stanford University, Palo Alto, CA (1966)

  28. Wachter, A.: An interior point algorithm for Large-Scale nonlinear optimization with applications in process engineering. PhD thesis. Carnegie Mellon University, Pittsburgh, Pennsylvania (January 2002)

  29. Liao, L.Z.: Optimal control approach for large scale unconstrained optimization problems (1995). http://www.citeseer.ist.psu.edu/liao95optimal.html

  30. Bellman, R.E.: Dynamic Programming. Princeton University Press, Princeton (1957)

    MATH  Google Scholar 

  31. Yakowitz, S.J., Rutherford, B.: Computational aspects of discrete-time optimal control. Appl. Math. Comput. 15(1), 29–45 (1984)

    Article  MathSciNet  MATH  Google Scholar 

  32. Liao, L.Z., Shoemaker, C.A.: Advantages of differential dynamic programming over newton’s method for discrete-time optimal control problems. Technical report. Cornell University (1993)

  33. Dreyfus, S.E.: Dynamic Programming and the Calculus of Variations. Academic Press, New York (1965)

    MATH  Google Scholar 

  34. Bryson, A.E.: Dynamic Optimization. Addison-Wesley, Menlo Park (1999)

    Google Scholar 

  35. Mayne, D.Q.: A second-order gradient method for determining optimal control of non-linear discrete time systems. Int. J. Control 3, 85–95 (1966)

    Article  Google Scholar 

  36. Jacobson, D.H., Mayne, D.Q.: Differential Dynamic Programming. Elsevier, New York (1970)

    MATH  Google Scholar 

  37. Gershwin, S., Jacobson, D.H.: A discrete-time differential dynamic programming algorithm with application to optimal orbit transfer. AIAA J. 8, 1616–1626 (1970)

    Article  MathSciNet  MATH  Google Scholar 

  38. Dyer, P., McReynolds, S.: In: The Computational Theory of Optimal Control. Academic Press, New York (1970)

    Google Scholar 

  39. Yakowitz, S.J.: Algorithms and computational techniques in differential dynamic programming. In: Control and Dynamical Systems: Advances in Theory and Applications, vol. 31, pp. 75–91. Academic Press, New York (1989)

    Google Scholar 

  40. Whiffen, G.J., Sims, J.: Application of a novel optimal control algorithm to low-thrust trajectory optimization. No. AAS 01-209, February (2001)

  41. Kraft, D.: On converting optimal control problems into nonlinear programming problems. In: Computational Mathematical Programming. Springer, Berlin (1985)

    Google Scholar 

  42. Stryk, O.V., Bulirsch, R.: Direct and indirect methods for trajectory optimization. Ann. Oper. Res. 37(1), 357–373 (1992)

    Article  MathSciNet  MATH  Google Scholar 

  43. Hull, D.G.: Conversion of optimal control problems into parameter optimization problems. J. Guid. Control Dyn. 20(1), 57–60 (1997)

    Article  MATH  Google Scholar 

  44. Biegler, L.T.: Efficient nonlinear programming algorithms for chemical process control and operations. In: IFIP Advances in Information and Communication Technology, System Modeling and Optimization, vol. 312, pp. 21–35. Springer, Boston (2009)

    Google Scholar 

  45. Taghavi, S.A., Howitt, R.E., Marino, M.A.: Optimal control of ground-water quality management: nonlinear programming approach. J. Water Resour. Plan. Manag. 120(6), 962–982 (1994)

    Article  Google Scholar 

  46. Spagele, T., Kistner, A., Gollhofer, A.: A multi-phase optimal control technique for the simulation of a human vertical jump. J. Biomech. 32(1), 87–91 (1999)

    Article  Google Scholar 

  47. Enright, P.J., Conway, B.A.: Discrete approximations to optimal trajectories using direct transcription and nonlinear programming. J. Guid. Control Dyn. 15(4), 994–1002 (1992)

    Article  MATH  Google Scholar 

  48. Betts, J.T., Erb, S.O.: Optimal low thrust trajectories to the moon. SIAM J. Appl. Dyn. Syst. 2(2), 144–170 (2003)

    Article  MathSciNet  MATH  Google Scholar 

  49. Bellman, R.E., Dreyfus, S.E.: Applied Dynamic Programming. Princeton University Press, Princeton (1962)

    MATH  Google Scholar 

  50. Colombo, C., Vasile, M., Radice, G.: Optimal low-thrust trajectories to asteroids through an algorithm based on differential dynamic programming. Celest. Mech. Dyn. Astron. 105(1), 75–112 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  51. Whiffen, G.J.: Static/dynamic control for optimizing a useful objective. No. Patent 6496741, December (2002)

  52. Whiffen, G.J., Shoemaker, C.A.: Nonlinear weighted feedback control of groundwater remediation under uncertainty. Water Resour. Res. 29(9), 3277–3289 (1993)

    Article  Google Scholar 

  53. Petropoulos, A.E., Russell, R.P.: Low-thrust transfers using primer vector theory and a second-order penalty method. In: AAS/AIAA Astrodynamics Specialist Conference and Exhibit, Honolulu, HI (2008). No. AIAA-2008-6955

    Google Scholar 

  54. Morimioto, J., Zeglin, G., Atkeson, C.G.: Minimax differential dynamic programming: application to a biped walking robot. In: SICE 2003 Annual Conference Fukui University, Japan, August (2003)

    Google Scholar 

  55. Chang, S.C., Chen, C.H., Fong, I.K., Luh, P.B.: Hydroelectric generation scheduling with an effective differential dynamic programming algorithm. IEEE Trans. Power Syst. 5(3), 737–743 (1990)

    Article  Google Scholar 

  56. Lin, T.C., Arora, J.S.: Differential dynamic programming for constrained optimal control. Part 1. Theoretical development. Comput. Mech. 9(1), 27–40 (1991)

    Article  MATH  Google Scholar 

  57. Ruxton, D.J.W.: Differential dynamic programming applied to continuous optimal control problems with state variable inequality constraints. Dyn. Control 3(2), 175–185 (1993)

    Article  MathSciNet  MATH  Google Scholar 

  58. Ohno, K.: A new approach to differential dynamic programming for discrete time systems. IEEE Trans. Autom. Control 23(1), 37–47 (1978)

    Article  MathSciNet  MATH  Google Scholar 

  59. Yakowitz, S.J.: The stagewise Kuhn–Tucker condition and differential dynamic programming. IEEE Trans. Autom. Control 31(1), 25–30 (1986)

    Article  MathSciNet  MATH  Google Scholar 

  60. Derbel, N.: Sur l’utilisation de la programmation dynamique différentielle pour la commande optimale de systèmes complexes. PhD thesis. INSA, Toulouse, France (March 1989)

  61. Patel, P., Scheeres, D.J.: A second order optimization algorithm using quadric control updates for multistage optimal control problems. Optim. Control Appl. Methods 30, 525–536 (2009)

    Article  Google Scholar 

  62. Giannessi, F.: Constrained Optimization and Image Space Analysis. Volume 1: Separation of Sets and Optimality Conditions, 1st edn. Mathematical Concepts and Methods in Science and Engineering. Springer, Berlin (2005)

    Google Scholar 

  63. Fletcher, R.: Practical Methods of Optimization, 2nd edn. Wiley, New York (2000)

    Google Scholar 

  64. Liao, L.Z., Shoemaker, C.A.: Convergence in unconstrained discrete-time differential dynamic programming. IEEE Trans. Autom. Control 36(6), 692–706 (1991)

    Article  MathSciNet  MATH  Google Scholar 

  65. Conn, A.R., Gould, N.I.M., Toint, P.L.: Trust-Region Methods. SIAM, Philadelphia (2000)

    Book  MATH  Google Scholar 

  66. Coleman, T.F., Liao, A.: An efficient trust region method for unconstrained discrete-time optimal control problems. Comput. Optim. Appl. 4(1), 47–66 (1995)

    Article  MathSciNet  MATH  Google Scholar 

  67. Gill, P.E., Murray, W., Picken, S.M., Wright, M.H.: The design and structure of a Fortran program library for optimization. ACM Trans. Math. Softw. 5(3), 259–283 (1979)

    Article  MATH  Google Scholar 

  68. Tang, J., Luh, P.B.: Hydrothermal scheduling via extended differential dynamic programming and mixed coordination. IEEE Trans. Power Syst. 10(4), 2021–2028 (1995)

    Article  Google Scholar 

  69. Courant, R.: Variational methods for the solution of problems of equilibrium and vibrations. Bull. Am. Math. Soc. 49, 1–23 (1945)

    Article  MathSciNet  Google Scholar 

  70. Birgin, E.G., Castillo, R.A., Martinez, J.M.: Numerical comparison of augmented Lagrangian algorithms for nonconvex problems. Comput. Optim. Appl. 31(1), 31–55 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  71. Bertsekas, B.P.: Constrained Optimization and Lagrange Multiplier Methods. Academic Press, San Diego (1982)

    MATH  Google Scholar 

  72. Niu, L., Yuan, Y.: A new trust-region algorithm for nonlinear constrained optimization. J. Comput. Math. 28(1), 72–86 (2010)

    MathSciNet  MATH  Google Scholar 

  73. Bertsekas, D.P.: Combined primal–dual and penalty methods for constrained minimization. SIAM J. Control Optim. 13(3), 521–544 (1975)

    Article  MathSciNet  MATH  Google Scholar 

  74. Dostal, Z.: Semi-monotonic inexact augmented Lagrangians for quadratic programming with equality constraints. Optim. Methods Softw. 20(6), 715–727 (2005)

    Article  MathSciNet  MATH  Google Scholar 

  75. Majji, M., Junkins, J.L., Turner, J.D.: High order methods for estimation of dynamic systems. J. Astronaut. Sci. 56(3), 401–440 (2008)

    Google Scholar 

  76. Park, R.S., Scheeres, D.J.: Nonlinear semi-analytic methods for trajectory estimation. J. Guid. Control Dyn. 30(6), 1668–1676 (2007)

    Article  Google Scholar 

  77. Murray, D.M., Yakowitz, S.J.: Constrained differential dynamic programming and its application to multireservoir control. Water Resour. Res. 15(5), 1017–1027 (1979)

    Article  Google Scholar 

  78. Gill, P.E., Murray, W., Wright, M.H.: Practical Optimization. Academic Press, San Diego (1982)

    Google Scholar 

  79. Kuhn, H.W., Tucker, A.W.: Nonlinear programming. In: Proceedings of 2nd Berkeley Symposium, pp. 481–492. University of California Press, Berkeley (1951)

    Google Scholar 

  80. Dennis, J.E., Heinkenschloss, M., Vicente, L.N.: Trust-region interior-point SQP algorithms for a class of nonlinear programming problems. SIAM J. Control Optim. 36(5), 1750–1794 (1998)

    Article  MathSciNet  MATH  Google Scholar 

  81. Rodriguez, J.F., Renaud, J.E., Watson, L.T.: Trust region augmented Lagrangian methods for sequential response surface approximation and optimization. J. Mech. Des. 120(1), 58–66 (1998)

    Article  Google Scholar 

  82. Lin, C.J., Moré, J.J.: Newton’s method for large bound-constrained optimization problems. SIAM J. Optim. 9(4), 1100–1127 (1999)

    Article  MathSciNet  MATH  Google Scholar 

  83. Ocampo, C., Senent, J.S., Williams, J.: Theoretical foundation of Copernicus: a unified system for trajectory design and optimization. In: 4th International Conference on Astrodynamics Tools and Techniques, Madrid, Spain, May (2010)

    Google Scholar 

Download references

Acknowledgements

This work was partially supported by Thales Alenia Space. The authors thank Thierry Dargent for support and collaborations, and Greg Whiffen for his valuable insight, feedback, and general introductions to DDP based methods.

Author information

Authors and Affiliations

  1. School of Aerospace Engineering, Georgia Institute of Technology, 270 Ferst Dr., Atlanta, GA, 30318, USA

    Gregory Lantoine

  2. Department of Aerospace Engineering and Engineering Mechanics, The University of Texas at Austin, 1 University Station C0600, Austin, TX, 78712-0235, USA

    Ryan P. Russell

Authors
  1. Gregory Lantoine
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryan P. Russell
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Gregory Lantoine.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Lantoine, G., Russell, R.P. A Hybrid Differential Dynamic Programming Algorithm for Constrained Optimal Control Problems. Part 1: Theory. J Optim Theory Appl 154, 382–417 (2012). https://doi.org/10.1007/s10957-012-0039-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10957-012-0039-0

Keywords

Search

Navigation