When something is observed it is always observed from some vantage point.
If two people are running next to each other it will not seem as if the other is moving, since their relative motion is the same.
Since the motion is really the same from any vantage point the reference point used to analyze physics can be chosen to be anywhere.
However, some reference points are more useful than others (depending on the situation).
For instance, with gravity- the lowest point in a certain situation could be ten meters above ground if an object is sitting on top of a roof, thirty two meters below ground if there is a hole, or at ground level.
Setting a convenient reference point will often make the manipulation of equations simpler though.
For instance, setting the reference point on the ground=0 level can make an integral easier because the bound of zero may cancel out terms.
Having zero at ground level may not always be convenient though, and always using zero to cancel out terms can be deceiving since it will not always work that way.