Mobile devices have become an integral part of 21st century lifestyle. From social networking and business to day-to-day scheduling and multimedia applications, smartphones and other portable handsets are now the go-to devices for interaction in the digital world. Currently, mobile devices typically utilise direct user interfaces such as touch screens, where interactions are performed directly by controlling graphical elements or controls on the interface. This project looks to bring device interaction out of the virtual world and into the physical world. With a 'free-gesture' approach, portable applications can break away from the virtual world, enabling the mobile platform to be harnessed as a physical augmented interface for musical performance, education, medical research and beyond.