My previous car was a Camry as well, with a V6. What model year is yours? I hated that car. It's the most boring appliance one could purchase/use for purposes of transportation. I hated it from the day I bought it, but at that time, finances decided my purchases. There is a world of difference between the two, and no, the Q50 is more than two Camry's put together. Difference is like night and day.
My Q50 is a RWD model, and you notice the difference in driving almost right away. With the Camry, it's always understeer, understeer, understeer. Even when making a right turn on low speeds on neighborhood streets, its always understeer. With the Q50, you can place the car precisely where you want it on the road, whether highspeed or low speed turns. And that exhaust growl, especially in Sport mode, is so intoxicating. I want to run up the RPMs just to listen to that exhaust, many times.
Once you drive a RWD car, it is very hard to go back to FWD. I say, if your budget does not allow it, then no, don't get it. A car, no matter how good, is not benefitting if you can't afford to drive it. But if you can afford it, go for it, you won't regret it at all.