State of the Common Lisp ecosystem, 2020
I think this article (of sorts) is definitely helpful for onlookers to Common Lisp, but doesn't provide the full "story" or "feel" of Common Lisp, and I want to offer to HN my own perspective.Disclaimer #1: I've been working professionally as a Common Lisp programmer---not as a contractor!---for the past decade. I have a vested interest in the language and hiring for it.Disclaimer #2: I am going to ignore commercial implementations of Lisp here, which provide very useful and advanced features, like GUI development, a user-friendly IDE, paid support, etc. [1,2]So let's get started. Common Lisp's best feature is that it allows you to be insanely productive at the "raw programmer" level. You can write, edit, and debug code very quickly and incrementally, and end up with safe & performant code.There's a price to pay: currently the best-in-class experience is still Emacs and SLIME (which come nicely packaged here [3]). As an Emacs fan, that's the best news, but to my fellow PyCharm/VSCode/vim users, it's terrible and alienating news. My colleagues who aren't Emacs users managed to learn just enough Emacs to be productive in a week, but they still frequently fired up their editor of choice in times of need.It really is worth underscoring that the Emacs+SLIME experience truly fits Common Lisp development like a glove, and the experience, in my opinion, is better than almost every mainstream editor environment out there.Common Lisp's worst feature is that it feels like just about everything imaginable has a catch. I don't mean "there's no free lunch", I mean that things just plainly don't feel cohesive or "100%" most of the time. To name a few examples:1. GUIs: If you want to make a traditional, native GUI using open source solutions, you're stuck with really goofy libraries that are non-obvious to get working. As the article points out, you have options. Lisp actually has a renowned framework called CLIM, but I consider the open-source implementation McCLIM [4] currently only principally useful to hobbyists and hackers.2. Deploying applications: Almost every implementation of Lisp has some way to create an executable. But very important aspects that real people care about in production are missing, inconsistent, or poorly documented. For example, almost no open source implementations of Lisp have first-class support for signing binaries on MacOS. Almost no open source implementations have a "tree shaker" to remove unnecessary cruft from the executable. Almost no open source implementations make building a shared library practical.3. Libraries: Many libraries don't do even usual things people might want to do. The linear algebra library MAGICL [8], for example, doesn't at the time of writing have a way to solve the matrix equation Ax=B. This isn't due to laziness of the authors or lack of foresight, but rather that it's a library that's just not used by enough people to see regular, high-quality contributions as an open-source project. I'm sure MAGICL solves problems for the authors, but the authors haven't taken it upon themselves to make a general, useful, and quasi-complete library for matrix programming in Lisp.These examples are just examples, maybe not even the top examples.There are many things I wish for Common Lisp, but there are two I think I wish most.First, I wish Common Lisp implementations put in work so that they could play nice with other programming languages. Google [11] recently came out with support for protobufs in Lisp, which is nice, but I feel something deeper is needed. I think Common Lisp implementations supporting building C ABI-compatible shared libraries would be an insanely big step forward; it'd mean that Lisp could feasibly used by every language out there. Right now, the closest we've got is Embeddable Common Lisp, an implementation of Lisp which makes embedding Lisp within C relatively painless, but as usual, it has many catches [12].The way I've coped is to produce stand-alone command-line applications, or to build servers with HTTP APIs. But it feels icky, especially if you're working with Python programmers who want to `import` stuff and not run servers just to get some code to work.Second, another thing that I constantly hope for in the Lisp world is for more "hyper productive" programmers to join it, or programmers whose livelihood depends on it. Of course, since Lisp is used by hobbyists, you see tons of hobbyist code. To be sure, a lot of this hobbyist code is perfectly fine. Usually it works, but it's just a tad incomplete. However, in my opinion, the worst thing about hobbyist code is that it usually doesn't do something useful.What does "useful" even mean? I won't claim to be able to define this term in a one-size-fits-all fashion, but "useful" to me is about getting practical computing work done. The further away from being concrete the library is, typically the less useful it is. For example, a typical Lisp programmer will have a penchant for writing a domain-specific language for parsing binary files (cool!), will open-source that code (cool!), but then nobody---including the author of said library---will actually use it to, say, write a parser for GIFs [5]. When somebody does come along to write a GIF parser, they're likely not going to use this general binary parsing framework, but hand-roll their own thing.In Lisp, it seems popular to solve meta-problems instead of problems, which is partly due to the fact that Lisp lets you think about problems at very high levels of abstraction using its advanced object system, the meta-object protocol, and macros.(One of my biggest "pet peeve" projects in Lisp, second only to "utility libraries", are documentation generator libraries. As soon as somebody figures out that documentation strings can actually be programmatically queried in Lisp, they invariably write a baroque "generator" that spits out HTML. I've never, not a single time, ever, used a documentation generator for doing real, paid work. I think one Lisp programmer I know uses it nicely is Nicolas Hafner, aka Shinmera, who uses a documentation generator simply to augment his long-form documentation writing. Staple [9] is one example library of his, where you can see some generated documentation at the bottom.)"Useful" also has to do with how a library is consumed. In the Common Lisp, a library like this [6] is typical. It's a bare page (be it on GitHub or otherwise) that provides no examples, no indication of dependencies, etc. Not all libraries are like this, but you run into it frequently enough.The Common Lisp ecosystem lacks a certain "go-getter" philosophy, needed to forge through "boring" work, that some other language ecosystems seem to have. To cherry pick one example, though I don't use it, Zig [7] comes out with interesting stuff all the time that's genuinely useful. Andrew Kelley, its main developer, is putting tons of hours into getting details around deployment right (e.g., cross-compilation). Little about Common Lisp prevents a motivated person from making equally productive-enhancing strides with the language, but I find that either (a) the interest isn't there or (b) the interest is there but the interest is for developing weird, esoteric stuff in Lisp.(My favorite example of a "productive stride" that happened in Lisp is the following. For context, people talk about all the time how difficult it would be to port a Lisp compiler to a new architecture. I myself have clamored for documentation on how to do it with SBCL. But, out of nowhere, some grad student named Charles Zhang came out with a port of SBCL to RISC-V. Not only did he port it, he's maintained it with 100s of new commits, making it more performant and less buggy [10].)Common Lisp is an amazing language purely from a practical point-of-view. As I said, to me, it's bar-none the best and most productive language to use if you want to "sit down and write code". The implementations of Lisp, like SBCL, are marvels. Lisp code, once you write it, will work forever (seriously, decades). The #lisp channel on Freenode is nice and helpful, and there are so many amazing people in the community. In Lisp, it's seamless to inspect assembly code and work with the world's most high-level, meta-object systems all at the same time. But the ecosystem mouthfeel is still off, and Common Lisp would greatly benefit from programmers obsessed with making the language more useful to themselves and others today.[1]: LispWorks: http://www.lispworks.com/[2]: Allegro CL: https://franz.com/products/allegro-common-lisp/[3]: Portacle: https://portacle.github.io/[4]: McCLIM: https://common-lisp.net/project/mcclim/[5]: There is a GIF parser though called SKIPPY! https://www.xach.com/lisp/skippy/[6]: MIDI: http://www.doc.gold.ac.uk/isms/lisp/midi/[7]: Zig: https://ziglang.org/[8]: MAGICL: https://github.com/rigetti/magicl[9]: Staple: https://shinmera.github.io/staple/[10]: Charles Zhang's SBCL commits https://github.com/sbcl/sbcl/commits?author=karlosz[11]: CL-PROTOBUFS: https://github.com/qitab/cl-protobufs[12]: Poorly documented, performance isn't very good, it's maintained by essentially one person, the rituals needed to use ECL-built libraries are more extensive than necessary, build times are insanely slow, ...