Apparently even setting the console to text mode doesn't help. Exiting X11 leaves the video card in a state that doesn't work with the display so I'm a bit lost about what's going on and how to fix it.