Dynamic generation of Python bindings for HPC kernels
Date
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Traditionally, high performance kernels (HPKs) have been written in statically typed languages, such as C/C++ and Fortran. A recent trend among scientists—prototyping applications in dynamic languages such as Python — created a gap between the applications and existing HPKs. Thus, scientists have to either reimplement necessary kernels or manually create a connection layer to leverage existing kernels. Either option requires substantial development effort and slows down progress in science. We present a technique, dubbed WayOut, which automatically generates the entire connection layer for HPKs invoked from Python and written in C/C++. WayOut performs a hybrid analysis: it statically analyzes header files to generate Python wrapper classes and functions, and dynamically generates bindings for those kernels. By leveraging the type information available at run-time, WayOut generates only the necessary bindings. We evaluate WayOut by rewriting dozens of existing examples from C/C++ to Python and leveraging HPKs enabled by WayOut. Our experiments show the feasibility of our technique, as well as negligible performance overhead on HPKs performance.