I think I was able to glue mine together. Reminder to be very careful when removing, shove the trim tool way down behind the console and you shouldn't have a problem. If you pry on the trim piece, you will probably have a broken trim piece.
I broke the outside trim piece as well, has anyone found a way to purchase the trim piece separately? From the toyota parts diagrams, I agree it looks like you need to purchase the entire assembly. Note that the part number you referenced is the non-sunroof version, the sunroof version is...