Do you have a physically separate, dedicated HD for Photoshop's scratch file? If not that Photoshop scratch is competing with the swap files of the OS and you could indeed run out of memory.
The rule of thumb I follow to figure out scratch space says to figure on 50 to 100 times the size of your largest file ever multiplied by the number of files you have open. I have seen the scratch file exceed 800 GB once, an admittedly rare occurrence, but it often exceeds 200 GB when stitching large panoramas and the like.
As an example—and stressing that I'm aware that others have even more scratch space than I do—I keep two dedicated, physically separate hard drives as my primary and secondary Photoshop scratch disks and a lot of GB free on my boot drive for the OS. I also have 16 GB of RAM installed.
Additionally, if you only have a single HD, i.e. your boot drive, you'd need it to be large enough to accommodate both the swap files of the OS as well as Photoshop's scratch.