Really depends on how much you want to future-proof on that VRAM buffer. Most games only require 2GB now, anyways - so, you'll still be over requirements quite well. Some games for me on my 4GB 960 are having VRAM usage over 3.5GB at 1080p on High to Ultra for me - such as Batman: AK, AC:U, & FC4. So, my buffer can handle the 4GB settings.
What this means is on your 970, you'll just have to turn down a few boxes in your in-game graphics options just so you avoid those silly stutters, framerate-drops, and nonsense like that. Shouldn't be the end of the world, TBH.
As long as you keep your games under that 3.5GB w/ that 970 (since your bus is 256-bit) - your card should blow mine out the bloody water on framerates + performance; i.e. just go check some benchmarks on the 970. On my 4GB 960, I have to sometimes lockdown framerates to 40FPS and stuff like that w/ EVGA Precision or MSI Afterburner if I want to keep Ultra or just jump down some settings from Ultra to Very High (if a game don't like EVGA Precision or MSI), just to stop my framerate from bouncing b/t 30-60 frames on a select few of the newer games. Or, if worse comes-to-worst - skip it, 30 frame lockdown via NVidia Panel (1/2 refresh on V-Sync) or in-game (like I did w/ Batman: AK).