First we said, "They're called adverbs because they attach to verbs."
Then we said, "Adverbs don't always attach to verbs."
The truth is, 'rules' in language are mostly approximations. The 'adverbs modify verbs' rule is one example of this.
Want to know how far back this goes?
'Adverb' is a Latin word that dates back more than 2000 years.
In Latin, ad means 'towards' and verb means 'verb, or word': which is how we get the definition for adverb to be 'added to the verb'.
But... since in Latin verb was a more general word, it kind of means something more like 'added to... a word'. Which is pretty vague.
So the fluid and flexible nature of adverbs goes back thousands of years.