I have a script that does some webscraping for news and then uploads the info I scrape into a PostgreSQL DB in RDS. My question is regarding the preferred method for ensuring that the same news article is not recorded more than once.
Each time I scrape, the web scraper returns 40 news articles. I have it set so that each article + headline is added to a table where the headline column has a UNIQUE key constraint. So I have 2 options in order to make sure that each article is only recorded once:

Use a simple try and except to try to insert every article + headline into the table -- error is returned if the headline already exists but it is ignored.
Or, I can query for the 40 most recently added articles in the database, compare their headlines to the ones I pulled, and then only insert those that aren't already in the database.

My question is: which one would be better performance-wise? My guess is that with a low number of articles number 1 would be better but as the number of articles increases it would be better to use number 2, is that correct?
Identifying the duplicates locally to the scraper will be faster than making a round-trip to the database to do so, provided it isn't done in a dumb way.  But the difference is very unlikely to be meaningful, compared to the overhead of doing the scraping in the first place.
But if the scraper has a limited memory for headlines, you will need to have a catch-and-ignore capability anyway.
PostgreSQL Unique Index performance