Optimizing the microstructure in open-die forgings using reinforcement learning

Optimizing the microstructure in open-die forgings using reinforcement learning

REINISCH Niklas, BAILLY David, HIRT Gerhard

Abstract. The open-die forging process can produce large workpieces with excellent material properties that can be used for heavy-duty applications like turbine shafts. The mechanical properties result from the microstructure, which in turn directly results from the process route. Since in open-die forging processes commonly hundreds of unique forming operations are carried out, numerous process routes lead to the same final geometry but produce different microstructures. This is why the prior design of an optimal pass schedule is essential to ensure good mechanical properties of open-die forgings. In the past reinforcement learning (RL) was already used to design optimized pass schedules for open-die forging that achieve the desired geometry, utilize the available press force, and reduce the number of passes. Furthermore, the design of a single pass schedule only took a few seconds which creates opportunities for the use of RL in control systems e.g. for the microstructure. This is why in this publication an existing RL algorithm is extended so that microstructure can be included in the optimization. Within this process, a microstructure model is integrated into the RL algorithm and the reward function (defines the goal of the training process) was extended in two steps to also rate the achieved average grain size continuously dependent on the temperature of the workpiece. In addition, the RL implementation was changed to ensure the production of the desired final geometry leading to a decrease in the complexity of the optimization problem. Thus, both an improvement of the designed pass schedules and a significant reduction of the training time of the RL algorithm was achieved.

Keywords
Open-Die Forging, Reinforcement Learning, Process Optimization, Grain Size

Published online 4/19/2023, 10 pages
Copyright © 2023 by the author(s)
Published under license by Materials Research Forum LLC., Millersville PA, USA

Citation: REINISCH Niklas, BAILLY David, HIRT Gerhard, Optimizing the microstructure in open-die forgings using reinforcement learning, Materials Research Proceedings, Vol. 28, pp 2061-2070, 2023

DOI: https://doi.org/10.21741/9781644902479-221

The article was published as article 221 of the book Material Forming

Content from this work may be used under the terms of the Creative Commons Attribution 3.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.

References
[1] Information on https://www.sms-group.com/de-de/plants/open-die-forging-presses
[2] Information on https://easy2forge.com/
[3] A. Esteso, D. Peidro, J. Mula, M. Díaz-Madroñero, Reinforcement learning applied to production planning and control, Int. J. Prod. Res. (2022). https://doi.org/10.1080/00207543.2022.2104180
[4] C. Li, P. Zheng, Y. Yin, B. Wang, L. Wang, Deep reinforcement learning in smart manufacturing: A review and prospects, CIRP J. Manuf. Sci. Technol. 40 (2023) 75-101. https://doi.org/10.1016/j.cirpj.2022.11.003
[5] O. Gamal, M.I.P. Mohamed, C.G. Patel, H. Roth, Data-Driven Model-Free Intelligent Roll Gap Control of Bar and Wire Hot Rolling Process Using Reinforcement Learning, Int. J. Mech. Eng. Robot. Res. 10/7 (2021) 349-356.
[6] J. Dornheim, N. Link, P. Gumbsch, Model-free Adaptive Optimal Control of Episodic Fixed-horizon Manufacturing Processes Using Reinforcement Learning. Int. J. Control Autom. Syst. 18 (2020) 1593–1604
[7] C. Scheiderer, T. Thun, C. Idzik, A.F. Posada-Moreno, A. Krämer, J. Lohmar, G. Hirt, T. Meisen, Simulation-as-a-Service for Reinforcement Learning Applications by Example of Heavy Plate Rolling Processes, Procedia Manuf. 51 (2020) 897-903. https://doi.org/10.1016/j.promfg.2020.10.126
[8] N. Reinisch, F. Rudolph, S. Günther, D. Bailly, G. Hirt, Successful Pass Schedule Design in Open-Die Forging Using Double Deep Q-Learning, Processes 9 (2021) 1084. https://doi.org/10.3390/pr9071084
[9] A. Tomlinson, J.D. Stringer, Spread and elongation in flat tool forging, J. Iron Steel Inst. 193 (1959) 157-162.
[10] M. Wolfgarten, D. Rosenstock, F. Rudolph, G. Hirt, New approach for the optimization of pass-schedules in open-die forging, Int. J. Mater. Form. 12 (2019) 973-983. https://doi.org/10.1007/s12289-019-01471-w
[11] D. Recker, M. Franzke, G. Hirt, Fast models for online-optimization during open die forging, CIRP Annals – Manuf. Technol. 60 (2011) 295–298. https://doi.org/10.1016/j.cirp.2011.03.142
[12] K. Karhausen, R. Kopp, Model for integrated process and microstructure simulation in hot forming, Steel Res. 63 (1992) 247-256. https://doi.org/10.1002/srin.199200509
[13] F. Rudolph, M. Wolfgarten, V. Keray, G. Hirt, Optimization of Open-Die Forging Using Fast Models for Strain, Temperature, and Grain Size in the Context of an Assistance System, Forming the Future. The Minerals, Metals & Materials Series (2021) 1145-1159.