The guys discuss their memories of being denied to a special garden or moments where they didn't fit in