I think you need to create your own lobby/room first, then you send an invite via the usual PS3 home menu, although I just noticed you are on XBox so no no idea if the message function is in the same place, but I'm pretty sure I needed to create the lobby first anyways.