Maybe Joseph worried that other people would think he was a wimp, or crazy, or foolish for heading to Bethlehem with Mary. Scripture says otherwise:
Matthew 1:19, 24 (ERV)
Joseph, was a good man . . . he did what the Lord’s angel told him to do.
All that matters. Lord, is what you think. May my Bethlehem journey be both good and obedient in your eyes.