I just tried what you’re describing on a Moto G Stylus, and didn’t have any trouble adjusting the volume.
Are you doing this while the voicemail is playing??
When you push the volume-down button on the side of the phone, it’s the slider with the “phone” icon that you’ll want to adjust. The speaker icon, is, however, meant to be very loud, so even at a low volume, it’s still fairly loud, but I can definitely tell a difference between maximum and minimum volume levels.
If you keep having trouble with it, let us know, and if you can think of any other details that might help us understand what you’re experiencing, maybe we can figure it out together.