Is it necessary to click the "safely remove usb" thing?

it kept saying that it was still in use, but i had completely no programs open. i even checked the task manager. so i went to device manager & chose "optimize as quick removal' it says you can disconnect the device safely w/o using the safely remove hardware icon. would anything happen to my hardware that i ejected?