Question:Why are cavity walls built?
Brick conducts heat fairly well.
If there is only a single layer wall, then the heat inside the home is conducted through the wall to the outside world fairly efficiently.
Since you do not want to pay to heat your garden, it is better for your heating bill if you can reduce heat loss by conduction through the walls of your house.
Cavity walls separate the inside of the house (where you are producing heat to keep you warm) from the outside of house (where the cold environment acts as a heat sink) by putting an air gap between the layers of brick.
Air is a gas and all gases are poorer conductors of heat than solids.