You seem to be conflating a caste system with serfdom- a caste system doesn't necessarily mean you're tied to the land. It simply tends to restrict you to a particular role or set of roles within society. Especially in pre-19th Century colonialist models it certainly wasn't necessarily the peasants who tended to set out looking for a better life but what we'd call middle to lower middle class people. If you look at North America, for example, they encouraged poorer types to go over as indentured servants initially but that idea got dumped in favour of using slaves which is why the American ethos was built around the small landowner or the small business owner- poor people certainly didn't have a great role in that idea.
A caste system wouln't necessarily change that- it would still be the lower middle classes (small independent farmers and businessmen) who would be going out to settle.
Having said that, serfdom certainly didn't seem to stop the Russian colonisation of Northern Eurasia- and I'd argue that that was just as much a colonial process as the settlement of the Americas or Australia.
No, I'm not confusing the two. If you have a set role in society and the higher-ups in that society have no intention of changing that or allowing for it to happen, then you can't go and just become a sailor or something. For colonisation itself it would simply mean carrying over people tied to a certain function and dumping them somewhere else, but you would still need a constant supply of sailors (not necessarily experienced captains, but low-rank sailors). A society that has a caste system that accomodates a more or less small scale sailing enterprise would have to change a lot to accomodate something like sailing all over the world.
I don't know the specifics of Russian colonisation, but I imagine that, being convenient to move the serfs elsewhere, the land they were tied to was changed by people who could technically do that. A caste system, as I say above, does allow for more geographic flexibility but not the flexibility of functions required. Not to mention that for many people the reason to join and support such an enterprise was that they expected they could have a better life or, if they died, their families could. In later centuries people would move overseas from Europe because they wanted to get rich in the colonies and return, too. A caste system would be more restrictive in this regard as well.
That said, of course that perhaps if it was convenient for the higher castes they could contrive to change it somewhat, but I assume that would eventually look less like a rigid caste system without much change until the present day.