This phrase sounds like it has its origins in crosswords but it originated from the early days of radio broadcasting of football matches.

So that listeners could follow the match more easily, a diagram of the pitch was published in the radio programme.

The diagram was divided into numbered squares and of course, the numbers started at 1.

Nowadays, the phrase means ‘back to where one started’.