Digital twinning of well construction operations for improved decision-making

Well construction is a highly technical, inherently unpredictable, and non-holonomic multi-step process with vast state and action spaces, that requires complex decision-making and action planning at every step. Action planning demands a careful evaluation of the vast action-space against the system’s long-term objective. Current human-centric decision-making introduces a degree of bias, which can result in reactive rather than proactive decisions. This can lead from minor operational inefficiencies all the way to catastrophic health, safety, and environmental issues. A system that can automatically generate an optimal action sequence from any given state to meet an operation’s objectives is therefore highly desirable. Moreover, an intelligent agent capable of self-learning can offset the computation and memory costs associated with evaluating the action space. This dissertation details the development of such intelligent planning systems for well construction operations by utilizing digital twinning, reward shaping and reinforcement learning techniques. To this effect, a methodology for structuring unbiased purpose-built sequential decision-making systems for well construction operations is proposed. This entails formulating the given operation as a Markov decision process (MDP), which demands carefully defining states and action values, defining goal states, building a digital twin to model the process, and appropriately shaping reward functions to measure feedback. An iterative method for building digital twins, which are vital components of this MDP structure, is also developed. Finally, a simulation-based search decision-time planning algorithm, the Monte Carlo tree search (MCTS), is adapted and utilized for learning and planning. The developed methodology is demonstrated by building and utilizing a finite-horizon decision-making system with discrete state- and action-space for hole cleaning advisory during well construction. A digital twin integrating hydraulics, cuttings transport, and rig-state detection models is built to simulate hole cleaning operations, and a non-sparse reward function to quantify state-action transitions is defined. Finally, the MCTS algorithm, enhanced by a well-designed heuristic function tailored for hole cleaning operations, is utilized for action planning. The plan (action sequence) output by the system, results in significant performance improvement over the original decision maker’s actions, as quantified by the long-term reward and the final system state