The origin of this is Chaza”l.
When the Kohein Gadol finished the ahvoideh on Yom Kipur and he was going home, there was a massive crowd accompanying him and people where trying to get close to him to kiss his hand. Many times he couldn’t get home till midnight.