I've been watching plenty of Markiplier's Let's Plays of horror games lately, and at some point I caught myself thinking that those aren't really scary. Yes, they have jumpscares, yes, they have the darkness, and typical horror music and so on, but they aren't scary. They don't terrify, don't make me grab to my seat and look around cautiously.
My reaction may be dulled by the fact that I'm watching someone else play the game and audiocomment over it (in case of Markiplier, with a small webcam segment as well). Even then, watching Birgirpall play through Alien: Isolation while exchanging jokes with Banzaii made me engaged into the game process even though I wasn't in any part involved in it. The game made me care about it even though I was behind two screens, and through that care, I was able to feel the horror it was supposed to represent.
I have a few ideas on the matter and I wonder what everyone else have uncovered so far.
First of all, it's the feeling of helplessness even while being in control of your character and their actions. In Alien: Isolation, you're hunted by a giant alien who's so much more powerful than you and can tear you apart if you make the wrong move. As you move through the game, you find tools to either distract the alien or even fight it enough to make it go away for a time. Then, the androids at the station get unconditionally aggressive towards the character and will hunt you down unless you escape or kill them, while displaying no emotions whatsoever, which leads me to another point.
The atmosphere. It's very hard to nail, and I'm not certain what exactly that makes it truly mortifying. Apparently, the darkness plays a lot into it, which is most of the horror games are set either at night or underground/inside. What I believe to be true is the game world's lack of care for the character: the fewer helpful tools and bonuses there are, the more helpless you feel, the more scared you feel. So, scarcity? Does darkness matter that much? Can you make a broad-daylight horror that would terrify the player?
Sound isolation can be important. Absolute silence terrifies more than sad violins or the constant noises in the corner, for then you have no idea what's coming. I've never seen a game that utilised that.
Jumpscares are oddly satisfying to experience, but are they important to the horror? Many games do good with it (the most recent example I've seen being Terrorift, here played by Markiplier using Oculus Rift), just as many simply place a thing that's not terrifying at all in front of you.
How about subtlety? It's not as simple to implement, but if done correctly, it can add the subconscious "Something's not right here" feeling while preserving the overall unbroken tension, to break it harder later.