It's the latter - buildings remaining heated during the night.

Even if the heating is off, you've also got all those server rooms, which are being kept cool. All the heat that's being exchanged there mounts up. Plus there's heat leaking from buildings, heat from cars and buses being in higher concentrations, and so on.

Basically, the city - that's the whole of central London, not just The City - is about one to two degrees warmer than the surrounding suburbs. (And the suburbs themselves are a degree or so warmer than teh surrounding countryside, as I understand it.)

Microclimates like this are fascinating stuff. :-)
http://en.wikipedia.org/wiki/Climate_of_London