Question: This question is rather conceptual and I just need some clarifications.

Let's say on your linux server you've something like ufw installed or windows firewall on windows server. And you have layers of networking devices like routers in between which will have their own firewalls. Why do you still need a hardware firewall as part of your infrastructure?

Answer: 
Not that long ago, typical designs forced all traffic through one big firewall as a security control. That is an advantage: by being at the Internet gateway, the network team can enforce a security control even if they don't manage every device. By necessity, such a firewall needed to be big with expensive custom hardware.

Now, it is possible to categorize traffic based on threat and importance, and distribute the packet filters. Perhaps send YouTube traffic direct to the Internet, it is already encrypted and possibly not mission critical. And in the data center or cloud, a distributed firewall on every compute host scales up the filter capacity, making improved segmentation possible. Both ease off on the demand for enormous hardware firewalls at the perimeter.

Also, open source software on commodity hardware has changed some of the big firewalls into software products. 25 Gb Ethernet is cheap, why not run the firewall on a server like other services? Vendors will still sell hardware appliances, especially on the large size, but the argument that their hardware is magic might not be as compelling.

Certainly you would want a defense in depth approach to security. Many components to consider: end point security agents, strong user authentication, network segmentation, encrypted data in transit, host level firewalls, perimeter firewalls, updating software on all the things. But merely the fact that a firewall has a special sauce ASIC in it doesn't necessarily define it.
As far as I am concerned its a matter of belt and braces. A hardware firewall helps to keep the bad guys out of your network. UFW and the like help you keep the bad guys out of your server if they do get onto your network. Generally routers are routers first and foremost and firewalls a distant second. They are better than nothing, but not a substitute for a proper firewall.