Only the *observable* Universe was concentrated into a point at the
time of the Big Bang, not the entire Universe.  The distinction
between the whole Universe and the part of it that we can see is
important.

We can see out into the Universe roughly a distance c*t, where c is
the speed of light and t is the age of the Universe.  Clearly, as t
becomes smaller and smaller (going backward in time toward the Big
Bang), the distance to which we can see becomes smaller and smaller.
This places no constraint on the size of the entire Universe, though.