I mean the fictional assumption is that it was a single group that wandered thousands of miles across continents; it was many generations that would settle somewhere, with subsequent generations migrating and spreading slowly, leaving behind traces like gravesites and settlements and the like; combine that with accurate dating and you can in fact tell the direction of travel.