Larry was born albino, and for this reason his parents, Smokey and Mavis, sold him to the circus. Smokey later comes to regret this and hopes to find his son, but knows he probably never will. Ironically, Frank has an encounter with him before ever meeting Smokey, at the employment office with Phineas and the other out of work circus workers.
In Season 3, it's revealed Smokey has known where he lives for some time but won't meet him because he's still repulsed by his son's albinism. It was also revealed that Smokey and Mavis conceived Larry in a laundromat.