Kramer is a musician, composer, and record producer whose creative ideas can be found all over the American cultural landscape. He is known as the founder of the record label Shimmy-Disc. He was a member of  New York Gong, Shockabilly, Bongwater, and Dogbowl & Kramer, and has toured with Butthole Surfers, B.A.L.L., Ween, Half Japanese, and The Fugs. As a producer, Kramer's notable work includes collaborations with Galaxie 500, White Zombie, GWAR, Daniel Johnston, and Urge Overkill.