Programming Language Development 101 - Prologue


I started off my professional career as a WordPress / PHP developer, transitioned to Laravel full-time and have since spent a lot of time building personal tools and hobby projects with "low-level" languages like Rust, C++ and C.

When I first started looking into programming language theory, design and development I felt like I gained a superpower. All of a sudden I understood the high-level concepts at the lowest levels of the tools we use every single day. When I look at my code now, I don't just see some words that tell the computer to do something. I see the processes that the language's interpreter or compiler goes through and how it meticulously transforms the text input it receives into meaningful output.

Take the following JavaScript program as an example.

class User {
    constructor(name) { = name

const Ryan = new User("Ryan")

Most of you will probably look at this and think "its just a class definition and instantiation of an object". You're not wrong at all. The JavaScript engine that executes this code however goes through a whole range of steps before it actually instantiates and creates a new object.

It has to tokenise our code, generate a structural representation of the program, convert that structure into some intermediate representation that the interpreter understands and only then does it actually execute the program.

The executed code needs to store our class definitions somewhere, allocate space in memory to store the object, figure out if an object is being used so that it can be garbage collected and that memory can be freed up.

Learning how these things are handled by the tools and programming languages we use everyday will likely change the way you perceive programming and help you appreciate the incredible work, craft and labour that goes into making those same tools usable by millions.

I'd like to help you reach that point in this new series, Programming Language Development 101. I'll go through the steps needed to develop a dynamically-typed language with similarities to JavaScript, Python and PHP.

By the end of the series, you'll hopefully learn how to write a tokeniser (lexer), parser and interpreter. To keep things simple and focus on the core concepts, we'll forgo a low-level language and implement everything inside with JavaScript.

I'll release a new post each week. You should have some time to go back and experiment a little, maybe get a headstart.

I hope you enjoy this series as much as I enjoyed writing it.

You might also enjoy...

Found this article useful? Share on Twitter!