Fix crash when pressing delete button in server browser and no server is selected.

A check for that was there, but was comparing an unsigned variable to
-1, which doesn't work.