Didn't Apple use to provide USB stick with OS X on it? I'm quite new to the Apple Mac world, and I've been told that some of the older macs used to include a USB stick to boot the OSX Mountain Lion.
I have a Macbook Pro with Mavericks, and there was no USB stick included. Did they discontinue this?
Yes, The USB recovery drive shipped first with a MacBook Air and Lion was also sold as a full retail OS on USB write only format instead of the retail DVD/CD that used to be available for purchase.

Apple discontinued this practice a few years ago and has started distributing OS X via the App Store exclusively as a digital download. You can make a bootable version of the installer from the App Store version if you have a USB stick handy.

There's a free, AppleScript-based program called DiskMaker X that will help you build a bootable USB stick for Mavericks. You'll have to download the installer from the App Store first. Just cancel the installation after the download and Mavericks will remain in your Applications folder for DiskMaker X to find and use.