Sorry if this is a really dumb question but I've been searching for ages and just can't figure it out.
So I have a question about unit testing, not necessarily about Python, but since I'm working with Python at the moment I chose to base my question on it.
I get the idea of unit testing, but the only thing I can find on the internet are the very simple unit tests. Like testing if the method sum(a, b) returns the sum of a + b.
But how do you apply unit testing when dealing with a more complex program? As an example, I have written a crawler. I don't know what it will return, else I wouldn't need the crawler. So how can I test that the crawler works properly without knowing what the method will return?
Thanks in advance!
The whole crawler would be probably tested functionally (we'll get there). As for unit testing, you have probably written your crawler with several components, like page parser, url recogniser, fetcher, redirect handler, etc. These are your UNITS. You should unit tests each of them, or at least those with at least slightly complicated logic, where you can expect some output for some input. Remember, that sometimes you'll test behaviour, not input/output, and this is where mocks and stubs may come handy.
As for functional testing - you'll need to create some test scenarios, like webpage with links to other webpages that you'll create, and set them up on some server. Then you'll need to perform crawling on webpages YOU created, and check whether your crawler is behaving as expected (you should know what to expect, because you;ll be creating those pages).
Also, sometimes it is good to perform integration tests between unit and functional testing. If you have some components working together (for example fetcher using redirect handler) it is good to check whether those two work together as expected (for example, you may create resource on your own server, that when fetched will return redirect HTTP code, and check whether it is handled as expected).
So, in the end:

create unit tests for components creating your app, to see if you haven't made simple mistake
create integration tests for co-working components, to see if you glued everything together just fine
create functional tests, to be sure that your app will work as expected (because some errors may come from project, not from implementation)
Python - unit testing