How should the UEFI System Partition be?

The UFI System Partition (ESP) is a required disk partition to boot on UEFI systems. Various Linux distributions recommends you set aside anything from 156 MiB to 513 MiB for the ESP. But how big does this special partition really have to be? and why do the different Linux installers recommend you set aside hundreds of Mebibytes for a partition they only install a couple of Mebibytes onto?