URL Feature representations I'm writing a bookmark classifier as a hobby/learning project. Currently I'm trying to decide on a feature representation.
I have two pieces of information:

*

*The name (meta title attribute)

*The URL

I'm primarily interested in whether there are any feature representations for URLs (or short length text in general) that perform better than bag-of-words.

URL's have the following format:

It tells you several things:

*

*Using https vs http tells you if the site is encrypted, this may, or may not be important information. Notice that you can have both https and http URLs pointing to the same website, so the fact that http exists does not mean that the website does not offer encryption.

*You would almost never see ports, so if you see one you can use a binary flag (yes/no) for it.

*In the domain name, the example above uses the .com top-level domain suffix. Those tell you something about the origin of the website, for example, .de is German country code, .edu is for educational purposes, there are suffixes as .mil or .gov for official, government pages, etc. Also keep in mind that there may be regional variations, e.g. .co.uk, or .gov.pl, where both parts of the suffix give you useful information. Notice however that to a great degree those may be misleading, for example, the .ai suffix in most cases would be used by a Silicon Valley AI startup, rather than someone coming from Anguilla.

*The domain itself can give you some clues about the content, for example, if you saw there wikipedia, amazon, or instagram, you would instantly be able to make a guess about the content. You could probably encode the most popular domains and keep the rare ones (johnnys-funny-cat-pics-blog.com) as the "other" category.

*Notice that the length of the domain is meaningful as well: the short ones are usually already taken, so they are either old, or someone bought them for a larger amount of money.

*The remaining /path/to/the/myfile.html can tell you something about the content of the page (e.g. login.php means that you can probably log in to something and penguins.html can be about penguins). In many cases, the words you see here would to some extent describe the content of the page. Treat them the same as any other words in other natural language processing tasks (bag-of-words, embeddings).

*The file extensions like .html, .php, .asp, etc can tell you something about the technology used.

*The ?key=value parameters would be used only in the pages that are more complicated than a static HTML-only page. They exist in pages that can receive parametrized GET requests. Their existence (yes/no) may be useful information in some cases. Their content (key=value pairs) can tell you what kind of information is send or received between the client and the server.

*The #Anchor links to a section of a webpage. Those would be used in places like Wikipedia, blogs, documentation, etc and you are unlikely to see them in non-articles pages (e.g. online store). Information of an URL having the anchor (yes/no) may be useful, and the anchor itself may tell you something about the content (e.g. if you saw #Population, this may be an article describing some country). If you have them in your data, consider them as well.

