Fear. How do you cause fear in a D&D game? Here is a bunch of people that have decided their careers should be going into the dark reaches of unpleasant places and fighting the things that go bump in the night, and the players understand this. How do you make them fear? Don't go for the cheap horror game/movie jump scares, that doesn't work in a tabletop RPG - the heroes will detect the monster(s) before they jump out, and it's never as dramatic as in a movie/videogame. Don't rely on describing the gory, gruesome deaths the monsters inflicted on previous foes, it won't be as vivid in the mind of your players as it is in yours. You can (and should) still describe these scenes, if for nothing else then proving that the PCs are righteous and the monsters are beyond the pale, just don't expect them to have any effect on the players. No, you have to work on the players.
Why even make the players fear for their characters' demise? If the game is too safe, if the PC death is a rare possibility, if the worst thing that could possibly happen is a fumble every once in a great while, there is no tension in the game. But every fight cannot be an impossible challenge, you have to modulate them - some challenges must be easy, a good cardio warmup to get the muscles loose and the blood pumping; some challenges must be hard, knuckles bitten as the initiative counts down as the players wait to see when the baddies go; and some challenges must be impossible, but obviously so, so that the players feel that stab of fear all the while they are beating that hasty retreat to safety (because violence may the answer to many problems, it shouldn't be the only answer, and talking it out doesn't always have to be an option, running away is perfectly acceptable).
My players are cautious, to the point that I thought that today, halfway through their first dungeon, they were going to cut and run and leave the other half unexplored. They didn't, but I have performed a psy-warfare campaign on them from before we even ran Session Zero, and it is obviously working.
-I made sure to mention to my players (one here or there is enough, you don't have to tell them all, separately or as a group) that there is a monster in the first dungeon that I didn't bother to stat out, merely put in my notes "if the party fights this, they die". That one player, knowing this deadly encounter is somewhere ahead, will work as your unwitting agent and caution the others that something horrible is out there.
-I also put in my intro to the campaign document multiple places the phrase "combat is deadly and the consequences of failure are dire". It subtly works into their brains
-In my house rules, I state that we are using the Lingering Injuries table from the DMG - "the consequences of failure are dire".
-And the inclusion of the aforementioned critical/fumble table became a lot more important in the players' minds when the first goblin ambush they hit managed to nigh kill one of the PCs with it (you may have to fudge this roll, but I lucked out and got a nat 20).
Am I trying to kill the PCs? No. I have even fudged a few rolls, especially in the "easier" fights (there is no glory in dying to a low CR monster), and not used every bit of tactical acumen I possess, but when they face the big bads, it'll be for keeps. No fudging, no throwing weak strategies at them, all the options in the stat block will be used. This is a fantastic little bit of video I've run across that is entertaining and a good explanation of why fudging die rolls at times is better than not.
As for the rules, we're all getting more familiar with them. Making my players read from the book the full text of each spell as they cast it or special ability as they use it is making us all familiar with the rules and especially letting everyone know what options there are (and aren't) in 5e rules. Like charging - in older editions (3e for me, it's the one I've played the most before now), anyone could make a movement based attack to damage and/or push their opponent around the map, yet 5e only allows it if you've taken the Charger feat. I've encouraged all of the players playing humans to not take feats (I get that they were popular in 3e but they're not really well implemented in 5e, so I fail to see the point of them in this edition), so I think I'm going to house rule in charging for everyone. I'm kicking myself as I type this that I didn't decide to allow it on the spot when it came up earlier today, but I'm definitely not the perfect DM.