How do I get the XFX R6770 Core Edition video card to work with my Mac Pro? I just bought a used Mac Pro (Mac Pro 1,1). The former user had bought an XFX R6770 replacement card for the NVIDIA 7300 (original) card. He hadn't installed it yet, but assured me that this was the recommended card from a Genius Bar employee.  

However, I installed the card, plugged in the PCI-E power cable, and...nothing. When I open up the Expansion Slot Utility (/System/Library/CoreServices/Expansion Slot Utility.app), it shows that in the first PCI slot, there is an "ATI VGA-Compatible Controller Card."  

I'm really not a hardware person at all. I'm completely lost. Any suggestions on how to get this card working?
Unfortunately, OS X Lion does not have driver support for the XFX R6770 Core Edition video card. You will have to upgrade to OS X Mountain Lion for the graphics card to work.
Since your MacPro (1,1) is &quot;officially&quot; unable to upgrade to Mountain Lion, you will have to &quot;trick&quot; your MacPro into thinking it is actually a MacPro(2,1). The reason for this is because a MacPro(2,1) can have Mountain Lion on it.
To do this, you will have to follow this guide to change the kernel of your MacPro. I have summarized the procedure of the guide if you prefer, so continue reading.
Before Reading: Please look to the links at the bottom, which are links on how to do some of the things mentioned in this guide
Explanation
The first issue (lack of EFI64 on MacPro(1,1)) can be dealt with by using the Chameleon bootloader. Chameleon will emulate EFI64 and therefore load 10.8. Chameleon has to be booted by your Mac as if it were Windows. Apple calls this BIOS emulation “Legacy Boot”. Legacy boot will only boot from internal connectors like the SATA, so no USB or Firewire drives.
Requirements
This will require a spare HDD in addition to the drive that your OS is on.

Download 10.8 From the app store. (you will need to do this on a supported mac)
Download Xcode from Apple.
Download Chameleon-2.1svn-r192
Go into Disk Utility and partition your spare HDD. Click the options button and choose Master Boot Record. Choose two partitions, the first 1GB (named BOOT) and the second can use the remainder of the disk (named Installer). Both partitions are formatted Mac OS Extended Journaled.
Run the Chameleon installer.
Change the install location to the BOOT that you created earlier.
Click Customize here and choose Standard under the Chameleon Bootloader package.
In Terminal: run sudo bless --folder /Volumes/BOOT --file /Volumes/BOOT/boot --setBoot --legacy
Download this folder and place it in the root of BOOT.
Inside, add your serial to the SMBIOS.plist and in org.chamleleon.boot.plist set the default partition to 0,2 which is the installer partition.

Now to the installer.

Locate your 10.8 installer. Right-click and chose ‘show package contents’. Locate InstallESD.DMG in the SharedSupport folder. Double ckick on InstallESD.DMG to mount it.
Open Terminal and navigate to the Mac OS Install ESD drive. cd /Volumes/Mac\ OS\ X\ Install\ ESD/ should do it.
Still in terminal, you need to mount the hidden BaseSystem.DMG type open BaseSystem.DMG and hit enter.
Now open Disk Utility. Click on BaseSystem.DMG on the left, then click restore, then drag your installer partition into the destination field.
Click Restore. This will put everything but the packages and kernel on the installer partition. (You may need to rename your installer partition back to “installer” as it may now be called MacOS BaseSystem.)

Next steps are to copy mach_kernel from MacOSInstallESD to the Installer partition.

In Terminal navigate to the Installer ESD with: cd /Volumes/Mac\ OS\ X\ Install\ ESD then copy mach_kernel to the installer partition with cp mach_kernel /Volumes/Installer/mach_kernel. Have a quick look in terminal to see that it’s there with: cd /Volumes/Installer/ then ls. You should see it there.
Now in your installer partition there is a system folder, and in that is an installation folder, in there is a packages symlink. Delete it and the copy the packages folder from the MAC OS Install ESD.
Finally you need to open OSinstall.mpkg with flat package editor which is part of the XCode install.
Drag Distribution to your desktop and open it in text edit.
There is a section headed var PlatformSupportValues there is a list of board IDs in speech-marks and separated with commas. like Mac-F42D88C8,Mac-F2218EA9,Mac-F42D86A9 you need to add your mac’s board it to it. This is specified in the smbios.plist in the Extra as Mac-F4208DC8 so if you’re using that smbios.plist then add that value. close TextEdit and go back to the flat package editor. delete Distribution from the package and then drag the one you edited from desktop into flat package editor, then save the package.

The Ultimate Moment

Now re-boot and hold down Option to bring up the list of drives, choose boot, then franticly tap the up or down arrow key until the Chameleon list of bootable drives appears. Choose Installer.
(The disk with osx on should be the GUID partition scheme). When the installer has completed, Chameleon will keep trying to boot the installer so edit org.chamleleon.boot.plist.
Set the default partition  to your Mountain Lion partition. It will probably be 1,2 or 2,2.
After the installation had completed, I used nano to create an fstab file in /etc. This has two entries in it which stop the boot and Installer mounting when OSX boots so they dont appear in Finder.
The entries look like this for each Partition you want to hide:
UUID=39C933A6-E5A4-3CFB-8841-21C89B0EDF77 none hfs rw,noauto change UUID to the appropriate one. The UUID of a partition can be found in system report under Serial ATA.

Links and Resources

Create entries in fstab to prevent volumes mounting
Editing OSInstall.MPKG with Flat Package Editor
Copying mach_kernel
Mounting BaseSystem.dmg
Finding InstallESD inside Install Mountain Lion