I understand where you are coming from, but the context necessary for it to make sense is that you don’t have to operate following the assumption that the seasons work the same for each hemisphere, one actually has a colder winter than the other, so that would affect the global average with a bias for the colder one. Think of it this way: the average winter temperature for the Southern hemisphere is not as cold as the average winter temperature for the northern one, so when taking them both into account for the global average, the temperatures in the north are going to skew the results (the Southern winter is not enough to offset the high temperatures of the northern summer, so the global average of the earth in its entirety is colder during the northen winter)

As to why that happens, well you are right that distance to the sun is one of the biggest factors for temperature, but it’s not the only one. Global climate is a dynamic system, with a lot pf variables interacting with one another. Things like ocean currents and the amount of landmasses play a big role in regional climates. For example, thanks to the ocean currents Europe enjoys a milder climate than it would given its latitude. In this case the Antarctic cold is unable to go north due to being sealed by the Antarctic current, which encircles the continent. Also, the Southern hemisphere has a lot less landmass than the northen one, and water is a temperature regulator, so that affects the climate down here as well.