I think I have talked about all of this before in previous posts, but wanted to ponder on it more. As well as talk on into the next arc of the campaign. I had built the last side quest from pieces chosen from the Monster Manual, but with no real plans other than the quaggoth were chasing the troglodytes at the behest of the kuo toa, who's were praying to and serving an aboleth, and incidentally, said aboleth was using the kuo toa's odd ability to create their own gods through belief to strengthen its own power. The purple worm was added later when it became obvious they party needed a goad to drive them out of their comfort zone (cautious approach that was taking way too long). I really did not expect the party to fight the worm, only run from it, and I could move it around at a whim to guide them away or towards other things. But then they tracked it down and used a polymorph spell on it that the worm failed to react in time and failed its save against, and suddenly the party is using it against their enemies.
It was brilliant! Utterly stupid but brilliant nonetheless. If the worm had acted before the spell was cast or it had made its save, the two party members involved in the tracking and spell casting part of the plan (the other three party members were well away from them) would surely have died. The three remaining party members would have had to continue on without the other two (no resurrections or the like in the party at the time) and utterly failed against the kuo toa and an upgraded aboleth. Or they could have tried to locate at least a little of their friends' bodies and gone back to the surface (by teleportation circle, if they remembered they had it, a long walk if they did not) to get them resurrected and then try again. The second option was also pretty much doomed to fail, because the kuo toa and aboleth would have been even more prepared for the party, and would not be playing nice. That is, if the purple worm did not simply TPK the lot of them in the first place.
With luck, however, they managed to pull it off. The lesson here is, as the DM, I got to write the beginning of this "story" (there are rumblings in the deep and the party needs to take care of it), the ending (the party succeeds if they kill or drive off the aboleth, or simply convince the aboleth to leave by getting the kuo toa to stop worshipping/serving it), and who was involved. Beyond that, setup was even easier as I decided to come up with maps on the fly, and since we were using Roll20 due to COVID-19 and a statewide isolate in place order, I was not terribly familiar with that program, so improvisation was the way to go. Yes, the party would have faced a much tougher threat if they had avoided the worm - the kuo toa want to capture you, usually to make you into their slaves, but in this case to give the party to the aboleth and let it mind dominate everyone into servitude - but going after the worm was much more precipitously dangerous as opposed to being merely dangerous for a longer period of time.
Therein lies this post's lesson - even as unscripted as you make your adventures, your party can still surprise you. And that's kind of the point of playing games like Dungeons and Dragons, the players face problem X, but this group may choose solution A, that group will waffle between solutions B and C, and yet another group will surprise you with solution Purple Ostrich. I have been having a lot of fun playing pre-generated adventures, The Shattered Gates of Slaughtergarde and Red Hand of Doom from 3e/3.5, and after the adventure I talked about above we have already had 3 sessions in the updated Against the Giants found inside Tales from the Yawning Portal collection of adventures, as I did not have to do much in the way of session prep. Yes, there was some prep work, especially transforming the 3e adventures into 5e-friendly fares, but very little deep investment of time. But as little as I had to do for the purchased adventures, for this little sidequest, I did amazingly little as well.
I had originally heard of the idea of using an adventure generator from the Web DM folks, as one of them was using basically a random table to make the games for his Spelljammer campaign. These side quests were very much like that, but I kept them as simple as possible - pick a monster or combination of monsters that sounded interesting, figure out a location that made sense to encounter them (or even made them more interesting), and then add that combo of monster and place into my game world. Shadows are terribly interesting if you pair them with a necromancer who has turned their generation of new shadows from the bodies of slain foes from hours into mere rounds, especially if they are also encountered in a subterranean realm where no sunlight can normally reach. A skull lord is a much higher level of challenge than my party was, but what if it could only exist on the Prime Material for a few rounds before being recalled? There were other ideas - some tied into a character's background (sometimes you just cannot avoid that one player who wants to write so much outside-of-the-game fiction that some of it will eventually worm its way into your game), others tied into other character's in between campaign downtime activities, but nothing screamingly unexpected and world-changing in terms of D&D adventures. Sometimes you throw curveballs at your players, and other times you give them the #2 combo they've always ordered from their local fast food restaurant - it may not surprise them, but what they get they will enjoy.
There you go, bunch of ideas, some funny stories, hope everyone enjoyed the ride.