By default Google Maps is used as directions provider for the directions link in the map panel and for the directions link added to the popuptext of each marker.
If you entered an address for your location, this address will be used for directions starting point, if you only entered latitude and logitude coordinates (and address field is empty), these coordinates are used as starting point for directions.
If you notice that your starting point on the opened Google Maps is incorrect, than is because Google by default always calculates the directions by using car as preferred transport method. So the nearest place to your location that can be reached by car is taken as starting point automatically.
Unfortunately this cannot be fixed on Google Maps, as this is not a bug but a feature 😉 – anyway there are several ways to deal with this issue:
– if you only rely on walking directions, you can navigate to Settings / directions settings / Google Maps directions settings and check the checkbox “Walking directions”.
– if that is not an option for you (roads for cars will be ignored with this direction), please navigate to Settings / directions settings / general directions settings and set a different directions provider like bing maps, which takes exactly the coordinates of your location and does not change the starting point.