I think it takes the right combination of elements to make a game that is more than just a pop-up scare game.
Dead Space had a lot of pop-up scare tactics where things would break through walls, vent covers, etc, suddenly. While this can be scary in the moment, it’s not a method to create a scary atmosphere. There is the lingering fear that something is going to pop out of every single vent shaft and floor grate, which goes to paranoia (which is good) but it takes more than that. Dead Space had good ambient sound. There were times when I wasn’t sure if a noise was coming from an enemy or was just ambient sound, adds good atmosphere.
The new AvP game could hardly be called a horror game, but playing as the marine there is unbundant amounts of fear. The motion tracker allows you to both see and hear something that tells you either a fan blade is moving or an alien is coming to rip your face off. The fact that the m-tracker allows you to hear, that wonderful ping ping ping sound effect, enemies when you can’t see them creates opportunities to scare the player with enemies that make noise and appear on the tracker but never engage the player. Knowing that there are enemies nearby, or was it just a physics prop moving?, and that you can’t see them creates both tension and paranoia.
Settings play a large role in it as well. While the Ishimura was a dark, foreboding place, it was overdone. Either it should have had power in places, or not. Opening a door and seeing pitch black would have been just as frightening, or more so because of the natural fear of the dark and unknown. Flickering lights are fine, where appropriate, but having them everywhere is a cheap excuse. Dynamic lighting, especially shadows, help create tension through expectation and, again, paranoia. Seeing a monstrous shadow and turning the corner to find nothing increases fear because of the unkown factor.
Music is also a big player in scary games. Games like Amnesia which do not rely on combat don’t use the often up-tempo fight tracks that games like Dead Space do. Music shouldn’t be absent, but it also shouldn’t take center stage. The music should play to the atmosphere of the moment, building tension and apprehension.
That’s just me though.