Naturalness and Bimodality of Software, and the practical use of Language Models

Abstract

After discovering, back in 2011, that Language Models are useful for modeling repetitive patterns in source code (c.f. The "Naturalness" of software <https://dl.acm.org/doi/10.5555/2337223.2337322>), and exploring some applications thereof, more recently (since about 2019) our group at UC Davis has focused on the observation that Software, as usually written,  is bimodal, admitting both the well-known formal, deterministic semantics (mostly for machines) and probabilistic, noisy semantics (for humans). This bimodality property affords both new approaches to software tool construction (using machine-learning) and new ways of studying human code reading. In this talk, I'll give an overview of the Naturalness/Bimodality program, some very recent work on "bimodal prompting", and finally some recent directions on evaluating the quality of code produced by large language models.