POTENTIAL SPOILERS FOR ALIEN: ISOLATION AHEAD.

I think a feeling of helplessness is a major factor. In Alien: Isolation, while you do get more items to try and fend off the Xenomorph, you never get strong enough to actually kill it. The fact that you can only fend it off momentarily and not destroy it gives an extra level of terror, since you know you have a limited amount of time to find a hiding place before it comes back after you've scared it off. Add to that the scarcity of items for crafting these distractions/weapons and you have the issue of whether you use the item now or hold it for later and just try to hide quickly.

Atmosphere is another huge part of it. Building tension is a difficult thing and horror games that do it well can achieve incredibly results. I agree with you and TheVenerableCain that a broad daylight game would be difficult to implement, because when you can see everything it's less scary. Darkness is a primal fear and the fear of the unknown or what might be there can add a lot of tension. I know in Isolation there were a couple points where parts of the scenery looked somewhat like the Xenomorph when viewed in half light, which did a great job of spooking you.

For a broad daylight game to work I think you would need some mechanic that made seeing things scary. Maybe something like in the Slender games where looking at the enemy causes difficulty with your vision and damages your health or something like the Eternal Darkness sanity effects, although I don't know how effective those would be in pure daylight.

Sound effects do a great job of building the atmosphere as well. Sitting in a hidey-hole or walking through a hallway hearing the Xenomorph stomp around could be terrifying. Isolation did a great job with the sound effects, especially when you would hear something walking outside or crawling through vents. It made you concerned that you could be dropped on at any time.

I think that Perspective is important too. The First Person perspective of Isolation lends itself to better tension building and horror because you can only see what you would actually see in real life. You can't switch to Third Person and see something sneaking up behind you. You have to use your "senses" (vision and hearing) and the somewhat unreliable motion tracker to get an idea of where anything is located. The motion tracker especially gives you a false sense of security at times since it only tracks things that are moving and when enemies stand still they don't show up. This adds in to your feeling of helplessness, since you don't have complete information about what's around you.

There are few games that I can remember that were truly as terrifying as Alien: Isolation was. I think it did a great job with all of the disparate elements needed to make a good horror game.