Multi-agent reinforcement learning for demand response and load shaping of grid-interactive connected buildings

Increasing electrification, integration of renewable energy resources, rapid urbanization, and the potential shift towards higher integration of electric vehicles and other distributed energy resources, such as batteries, represent challenges that the energy sector will have to face in the future. As these changes begin to happen, buildings will have more energy storage and generation capacity, which can create additional uncertainty and volatility, but also more flexibility if these systems are appropriately controlled. Demand response can enable consumers to reduce their energy consumption through load curtailment, shift their energy consumption over time to periods of higher renewable energy generation, or generate and store energy at certain times to provide the grid with more flexibility. Model-based control approaches, such as model predictive control, can provide near optimal control policies for demand response. However, they require to develop accurate models of the systems to be controlled, which is not always a cost-effective scalable option in complex, unpredictable or non-stationary systems. On the other hand, model-free control algorithms, such as reinforcement learning, have the potential to provide good control policies in a cost-effective manner. In this dissertation, I review the literature on reinforcement learning for demand response and control of urban energy systems, and introduce a novel multi-agent reinforcement learning algorithm specifically designed to be decentralized, scalable, and implemented in a demand response setting. This multi-agent reinforcement learning algorithm outperforms its single-agent counterparts and a baseline rule-based controller by more than 15% on an average of 5 different metrics: load factor, net electricity consumption, peak demand, average daily peak demand, and ramping. I test the controllers in CityLearn, an Open AI Gym environment I created for the implementation of single and multi-agent reinforcement learning for demand response. CityLearn is also intended to be used by other researchers and tackle another problem I observed in my review of the literature: lack of standardization and reproducibility of most research in RL applied to energy management in urban settings.