You can challenge your friends to a game only when they are in the same room as you. Open your friend list and look for the Dual Swords icon to challenge any player to a game.

If you have created your own game and are in the Lobby view, you can hover over the list of players in the room to invite a player to your game. When you hover over a player, a blue challenge button appears if they are eligible to play. Click this button to invite the player to join your game.