Finder remounts DMG files that I opened previously When I mount a .dmg file or open files in my drive (using text editor or other app), the Finder will automatically remount all .dmg images that I used before.

For example, I do something in DMG A, and unmount it and this will succeed.
However, when I open DMG B or open files in my drive (no matter what type of file), the Finder will remount the DMG A, so I have to unmount both DMG A and DMG B.

What kind of problem this is? and how can I resolve it?

Note: I installed TotalFinder on my machine—is this the cause of the problem?
This is strange behavior, but you might try checking Disk Utility. Load Disk Utility and look in the left sidebar. See if any DMG files are listed there. If they are, even if unmounted, click the DMG file and then the Eject button in the toolbar. If Eject is unavailable simply select the image and press the forward Delete key (Fn-Backspace on laptops). This should cause OS X to "forget" about that particular DMG file - at least until you mount it next.

If this is happening regularly - DMGs are always persistently remounting - then some extension is most likely causing it. If you removed this "TotalFinder" app by just dragging an .app file to the trash, it may not remove any extensions it installs into the OS that do those sorts of things. Look on the Internet - ideally on the software publisher's website - for instructions on completely removing the application, including all support and system files. Many apps that do install stuff deep into the system will come with an uninstaller or at least a shell script that will perform the necessary uninstallation commands for you.