The changes of the source illustrate the amount of conditional compilation that has accumulated in the past decades of Isabelle development. In the future, we will be able to proceed faster in just one direction, and pick up more advanced features of Poly/ML.
Already ten years ago, SML/NJ had little practical relevance for Isabelle users. It was occasionally used by tool developers to figure out difficult situations of type-inference, at a time when the SML/NJ compiler errors were better than the ones of Poly/ML. This has changed dramatically in recent years, with fairly good compiler errors and full IDE support in Poly/ML.
Performance of SML/NJ had always been a problem (after 1993): growing with the total heap size of Isabelle applications. This is inherent in the “CPS approach” of SML/NJ, see also this brief discussion on Stackoverflow.
The reason for giving up these tons of extra weight right after shipping Isabelle2016 are rather profane: in the routine tests of SML/NJ for the release, many basic Isabelle sessions (e.g. HOL-Library) no longer worked, running out of memory (within 32bit address space) after many hours.