Fragments of Carceral Memory weaves through interconnections between race, colonialism, and criminalization. I use a twofold approach to studying African-American, Caribbean, and French carceral narratives alongside penal heritage commemoration practices, with a primary focus on Guyane’s penal colony. Firstly, my analyses of anti-carceral literary texts includes dramas by Jean Genet and Lorraine Hansberry, an epic poem by Reinaldo Arenas, photo-texts by Eduardo Lalo and Patrick Chamoiseau, and a James Baldwin novel. Secondly, I address a discursive gap between colonial era prison tropes that characterize penal heritage sites and contemporary social movements aimed at abolishing prisons. This research conjoins a broad range of narrative work being created at individual, collective, and institutional levels—from the nineteenth century to the twenty-first—to explain how stories (both personal and imperial) must undergird public understanding of what I term a “colonial-carceral nexus.” Using disciplinary conventions in comparative literature to place Caribbean studies and carceral studies in dialogue, I trace how histories of targeted criminalization and pervasive imprisonment are recalled in literature, re-imagined through memory, and relived through legal systems. My research finds that institutional practices of penal heritage commemoration, in contrast to anti-carceral narratives authored by writers who have either been imprisoned or subjected to prolonged targeted surveillance, tend to view imprisonment in a favorable light. I propose some solutions to bridging this discursive gap and shifting the tenor of institutionally narrativized discourse. In conclusion, I contest the presumed incompatibility of carceral commemoration and prison abolition by raising a parting question: What if a tour in a former prison could become an educational opportunity for encouraging abolitionism?