In the second half your opponent hosts the game (assuming you 'created' it, other way round if you join a game). There will always be a little lag. Sometimes its worse, especially if your opponent is in a different country or has a rubbish connection.