Another crewman has died after the vessel - believed to be The Harvester - got into trouble on rocks at St David's Head on Thursday afternoon.
The rescued man was taken to hospital but was pronounced dead.
The search for the second man was scaled back on Friday with only routine patrols being carried out on the land on Saturday.
Wreckage from the boat has been found over a four mile (6.4km) area.
Five lifeboats, a coastguard helicopter using thermal image cameras and cliff teams have been involved in the search.
Speaking to BBC Radio Wales on Friday, Jim Phillips, from the RNLI in St Davids, said a member of the public on the coastal path saw the boat smashed on a rock with two people in the water at about 14:40 BST.
The vessel was about a mile (1.6km) out to sea near Abereiddi.
Mr Phillips said by the time rescuers got to the scene it had sunk and all that was left on the water's surface was debris.
Dyfed-Powys Police appealed for any witnesses to come forward.
The search for a fisherman missing after a boat sank in Pembrokeshire has been scaled back.