I see CSV as the only Upload format, does this mean that the format should exactly be the same as the Downloaded CSV for a list?

The uploaded file may not be the same as the downloaded file, it could be in a different format or different columns. But it is important to have the mapping fields for the uploaded format to be correct.