Question: We would like to get rid of the Tridion Broker as part of a project and publish content directly to a repository as part of a Storage Extension. 

My question is, is there any deep dependency that Tridion has on the Broker that might mean things work in a sub-optimal way, or gotchas that I should be aware of?

Update - We are going to be using Tridion purely as a content store which will push out JSON. There will be no functionality or any form of business logic/UI.

Answer: 
I will assume that what you mean with the Broker is the Tridion Broker database.

It is possible to publish everything to the file system, it includes, content, linking info, metadata and references, however this functionality is deprecated and will be removed in future releases, having said that, in future releases, linking, metadata and references will always be stored in the Tridion broker database.

If what you want is to override the storage layer and create a custom one, it will for sure affect key features like the Tridion CD API, Dynamic Linking and of course it won't be supported by either Customer Support or R&D.

Other key feature that depends on the Tridion Broker database is the Session Preview functionality.
1. Dynamic Linking is the big one as Rob mentions.
2. any DCPs embedded on Pages will render
 UCs/Tags.
3. Any code that queries CPs using Criteria/Query API will need to be gutted/replaced.
4. If you use P&P, then any personalized content will be rendered and filtered via User Controls/Tags, and this will need replacing
5. If you use UGC, same as #4.

When starting a fresh implementation Broker/no-Broker is a decision to be made, but once it's done and the site is live, you have a lot of work on your hands to get rid of the Broker.