He went for lunch every Wednesday with his younger sister Gisèle Casadesus.
He was briefly married with the singer Micheline Ramette and had a son with her, Frédérick Casadesus, born in 1959.
In 1948, he ran a jazz club with Freddy Chauvelot, the Club Saint-Germain, which staged musicians and singers such as Django Reinhardt, Juliette Gréco, and Boris Vian.
His father, Henri Casadesus, was a musician and composer, and his mother, Marie-Louise Beetz, a harpist.