My recollection is hazy, but I stayed in my selected college for entrance interviews. How or when I selected it, I cannot recall, but it was a critical part of the undergrad admission process, as it was the college that had the final say.
Generally, the intra-mural colleges - Balliol, New College, University, Jesus, Exeter, Lincoln, Trinity, Merton, Christ Church, etc are older and richer and right in the heart of Oxford. Then there's a number of red brick, Victorian colleges like Keble, Somerville, St Hilda's which are a little further out, and a couple of specialist ones that are basically monasteries.