Born in Brooklyn and raised in Massapequa, New York, Phantom grew up listening to his parents' jazz records and began playing drums by the age of ten. He took lessons with Mousie Alexander, who played with Benny Goodman, studying jazz and working through books by Jim Chapin and Ted Reed.

By the late 1970s, he was playing in bands with his childhood friend, bassist Lee Rocker. They soon joined forces with guitarist Brian Setzer to form Stray Cats.[ When performing with Stray Cats, Phantom did not sit behind a standard drum kit like most drummers, but instead opted to play in a standing position with a minimalist set-up of bass drum, snare drum, hi-hat, and crash cymbal.

During downtime from Stray Cats, Phantom played in swing, rockabilly, and jump blues combo The Swing Cats with Rocker and ex-Polecats guitarist Danny B. Harvey,[6] as well as rockabilly and glam rock project Phantom, Rocker & Slick with Rocker and