Why do programs run after I close them? I am more familiar with Windows and Linux where when I close the application window, the application is closed.
On Mac Big Sur, commonly, when I close an application, a small dot exists under the icon in the dock, indicating the program is still running. Why is this?
For example, while on Linux or Windows, when I close my sole Visual Studio Code window, the application completely closes. The process is completely gone. When I close vscode on mac, a small dot still exists in the doc, and when I open activity monitor, I can see the 'Code' process is still running.
On Windows or Linux an app's window usually is the app. However, macOS is different, as traditionally the app windows can be closed with the app still running (with few exceptions). The menu bar at the top of the screen shows you which app is currently active (there can be only one active app), and even if you close all windows of an app it will still be running and you will be able to control it with the menu bar.
The red circle closes the window and not the app, same is true for the ⌘ cmd+W shortcut (or ⌘ cmd+⌥ opt+W for closing all the windows of the active app).
To quit the app you can do one of the following:

pressing ⌘ cmd+Q to quit the active app
clicking on the app name in the top left corner of your screen (on your menu bar) and selecting Quit in the menu to quit the active app
right clicking (or ^ ctrl+click) on the app bar in the dock and selecting Quit
selecting the app in the Activity Monitor (Applications&gt;Utilities&gt;Activity Monitor) and using the Force quit button in its top left corner (useful for unresponsive apps)
pressing ⌘ cmd+⌥ opt+esc to open the Force Quit Applications window (useful for unresponsive apps)

You can read more about Mac window management here and about quitting apps here.