I've been thinking a little and I think the only way this is possible is if you know the maximum radius of the map around you. From there you can use similar triangle ratios and Pythagorean theorem and with a little bit of trig, figure out the adjacent side of both the yaw triangle and pitch triangle. That should get you the change in x and y you need. for the Z component, the height (opposite) part of the triangle should be the same on both triangles I think, else it's the pitch triangle.
Here are some pictures of the math I propose.
View attachment 2627
Sorry for bad drawings..
PS: to solve for the second variable use tan(Θ) = height/X, solve for one and plug it into the similar tri-equation in the picture.
Edit-
I did some scratch math on paper and the numbers aren't working out as well as I thought. I feel like I'm missing something here.
Another approach is tan(phi) = height / (x*y*sin(theta))
But the problem I can't completely workout is how to get at least one component to have a number.
Unfortunately similar triangles seems to be useless here.
So either I missed something crucial or it's simply impossible without an entity in the other end.