Contained in:
Book Chapter

Scheduling Optimization of Electric Ready Mixed Concrete Vehicles Using an Improved Model-Based Reinforcement Learning

  • Zhengyi Chen
  • Changhao Song
  • Xiao Zhang
  • Jack C. P. Cheng

Decarbonizing the construction sector has become an imperative global agenda, with electric machinery playing a pivotal role in realizing this objective. This research concentrates on devising an operational scheduling optimization method for electric ready-mixed concrete vehicles (ERVs) – a groundbreaking, eco-friendly intervention for the construction sector. We commence by outlining a systematic problem definition for the ERV operational process, considering the distinctive characteristics of electric vehicles and ready-mixed concrete (RMC) delivery tasks. The entire process is then conceptualized as a Markov decision problem (MDP), which enables sequential decision-making. We subsequently develop an enhanced model-based reinforcement learning technique, named parallel-masked-decaying Monte Carlo Tree Search (PMD-MCTS), for efficient resolution of the MDP. The entire system is authenticated via a real-world case study, and the PMD-MCTS's performance is juxtaposed against existing benchmarks. The results demonstrate the appropriateness of the proposed MDP formulation for tackling RMC delivery tasks. The PMD-MCTS algorithm and one of its ablation algorithms (PM-MCTS) have demonstrated superior performance compared to other benchmarks in either cost reduction or delay minimization, with PMD-MCTS requiring 30% less computation time than PM-MCTS

  • Keywords:
  • Electric vehicle,
  • Ready-mixed concrete delivery; Scheduling optimization; Model-based reinforcement learning; Monte Carlo Tree Search,
+ Show More

Zhengyi Chen

University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong

Changhao Song

University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong - ORCID: 0000-0003-0362-8445

Xiao Zhang

University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong

Jack C. P. Cheng

University of Hong KongThe Hong Kong University of Science and Technology, Hong Kong - ORCID: 0000-0002-1722-2617

  1. Bogachkova, L. Y., Guryanova, L. S., & Usacheva, N. Y. (2022). Decarbonization Trends in the Largest Post-soviet Countries and the Specifics of Their Inclusion in the Global Climate Agenda. New Technology for Inclusive and Sustainable Growth: Perception, Challenges and Opportunities, 77-88. DOI: 10.1007/978-981-16-9804-0_7
  2. Browne, C. B., Powley, E., Whitehouse, D., Lucas, S. M., Cowling, P. I., Rohlfshagen, P., . . . Colton, S. (2012). A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in games, 4(1), 1-43. DOI: 10.1109/TCIAIG.2012.2186810
  3. Gan, V. J., Chan, C. M., Tse, K., Lo, I. M., & Cheng, J. C. (2017). A comparative analysis of embodied carbon in high-rise buildings regarding different design parameters. Journal of Cleaner Production, 161, 663-675. DOI: 10.1016/j.jclepro.2017.05.156
  4. Hart, P. E., Nilsson, N. J., & Raphael, B. (1968). A formal basis for the heuristic determination of minimum cost paths. IEEE transactions on Systems Science and Cybernetics, 4(2), 100-107. DOI: 10.1109/TSSC.1968.300136
  5. Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735-1780. DOI: 10.1162/neco.1997.9.8.1735
  6. Huang, B., Boularias, A., & Yu, J. (2022). Parallel Monte Carlo Tree Search with Batched Rigid-body Simulations for Speeding up Long-Horizon Episodic Robot Planning. Paper presented at the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).
  7. Huang, S., & Ontañón, S. (2020). A closer look at invalid action masking in policy gradient algorithms. arXiv preprint arXiv:2006.14171. DOI: 10.32473/flairs.v35i.130584
  8. Karakatič, S. (2021). Optimizing nonlinear charging times of electric vehicle routing with genetic algorithm. Expert Systems with Applications, 164, 114039. DOI: 10.1016/j.eswa.2020.114039
  9. Lin, P.-C., Wang, J., Huang, S.-H., & Wang, Y.-T. (2010). Dispatching ready mixed concrete trucks under demand postponement and weight limit regulation. Automation in Construction, 19(6), 798-807. DOI: 10.1016/j.autcon.2010.05.002
  10. Lin, T., Lin, Y., Ren, H., Chen, H., Chen, Q., & Li, Z. (2020). Development and key technologies of pure electric construction machinery. Renewable and Sustainable Energy Reviews, 132, 110080. DOI: 10.1016/j.rser.2020.110080
  11. Liu, A., Chen, J., Yu, M., Zhai, Y., Zhou, X., & Liu, J. (2018). Watch the unobserved: A simple approach to parallelizing monte carlo tree search. arXiv preprint arXiv:1810.11755. DOI: 10.48550/arXiv.1810.11755
  12. Liu, Z., Zhang, Y., & Li, M. (2014). Integrated scheduling of ready-mixed concrete production and delivery. Automation in Construction, 48, 31-43. DOI: 10.1016/j.autcon.2014.08.004
  13. Liu, Z., Zhang, Y., Yu, M., & Zhou, X. (2017). Heuristic algorithm for ready-mixed concrete plant scheduling with multiple mixers. Automation in Construction, 84, 1-13. DOI: 10.1016/j.autcon.2017.08.013
  14. Luo, F.-M., Xu, T., Lai, H., Chen, X.-H., Zhang, W., & Yu, Y. (2022). A survey on model-based reinforcement learning. arXiv preprint arXiv:2206.09328. DOI: 10.48550/arXiv.2206.09328
  15. Olanrewaju, O. I., Edwards, D. J., & Chileshe, N. (2020). Estimating on-site emissions during ready mixed concrete (RMC) delivery: a methodology. Case Studies in Construction Materials, 13, e00439. DOI: 10.1016/j.cscm.2020.e00439
  16. Palaniappan, S., Bashford, H., Li, K., Fafitis, A., & Stecker, L. (2009). Carbon emissions based on transportation for post-tensioned slab foundation construction: A production home building study in the greater phoenix arizona area. International journal of construction education and research, 5(4), 236-260. DOI: 10.1080/15578770903355533
  17. Sinha, R. K., & Chaturvedi, N. D. (2019). A review on carbon emission reduction in industries and planning emission limits. Renewable and Sustainable Energy Reviews, 114, 109304. DOI: 10.1016/j.rser.2019.109304
  18. Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction: MIT press.
  19. Tan, S., Yang, J., Zhao, X., Hai, T., & Zhang, W. (2018). Gear ratio optimization of a multi-speed transmission for electric dump truck operating on the structure route. Energies, 11(6), 1324. DOI: 10.3390/en11061324
  20. Tong, Z., Jiang, Y., Tong, S., Zhang, Q., & Wu, J. (2023). Hybrid drivetrain with dual energy regeneration and collaborative control of driving and lifting for construction machinery. Automation in Construction, 150, 104806. DOI: 10.1016/j.autcon.2023.104806
  21. Turan, B., Pedarsani, R., & Alizadeh, M. (2020). Dynamic pricing and fleet management for electric autonomous mobility on demand systems. Transportation Research Part C: Emerging Technologies, 121, 102829. DOI: 10.1016/j.trc.2020.102829
  22. Volvo Trucks delivers the first heavy-duty electric concrete mixer truck to CEMEX. (2023).
  23. Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Langlois, E., . . . Ba, J. (2019). Benchmarking model-based reinforcement learning. arXiv preprint arXiv:1907.02057. DOI: 10.48550/arXiv.1907.02057
  24. Zhang, X., Zhang, J., Liu, Z., Cui, Q., Tao, X., & Wang, S. (2020). MDP-based task offloading for vehicular edge computing under certain and uncertain transition probabilities. IEEE Transactions on Vehicular Technology, 69(3), 3296-3309. DOI: 10.1109/TVT.2020.2965159
PDF
  • Publication Year: 2023
  • Pages: 739-750

XML
  • Publication Year: 2023

Chapter Information

Chapter Title

Scheduling Optimization of Electric Ready Mixed Concrete Vehicles Using an Improved Model-Based Reinforcement Learning

Authors

Zhengyi Chen, Changhao Song, Xiao Zhang, Jack C. P. Cheng

DOI

10.36253/979-12-215-0289-3.74

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Bibliographic Information

Book Title

CONVR 2023 - Proceedings of the 23rd International Conference on Construction Applications of Virtual Reality

Book Subtitle

Managing the Digital Transformation of Construction Industry

Editors

Pietro Capone, Vito Getuli, Farzad Pour Rahimian, Nashwan Dawood, Alessandro Bruttini, Tommaso Sorbi

Peer Reviewed

Publication Year

2023

Copyright Information

© 2023 Author(s)

Content License

CC BY-NC 4.0

Metadata License

CC0 1.0

Publisher Name

Firenze University Press

DOI

10.36253/979-12-215-0289-3

eISBN (pdf)

979-12-215-0289-3

eISBN (xml)

979-12-215-0257-2

Series Title

Proceedings e report

Series ISSN

2704-601X

Series E-ISSN

2704-5846

24

Fulltext
downloads

43

Views

Export Citation

1,307

Open Access Books

in the Catalogue

1,949

Book Chapters

3,290,448

Fulltext
downloads

4,134

Authors

from 860 Research Institutions

of 63 Nations

63

scientific boards

from 339 Research Institutions

of 43 Nations

1,150

Referees

from 345 Research Institutions

of 37 Nations