Invoice numbers come as #4 instead of #1004

I have invoice start number set as 1001, but they are coming out on the site and in email receipts as #1, #2, #3, etc.

How to fix?

Hello Bram,

Where did you see #1, #2, #3? I think they are order ID, not invoice ID. Please re-check to confirm.

Sincerely, Giang

Yes they are order ID, but how do I change them to Invoice ID?
It is very confusing, there are so many places to try and find the templates. ;-(
In "Customer Notification Email With Downloadable Products" I changed it to [ORDER_NUMBER] and it comes out "U9IJW5BQKC" on the email.
Then, on the PDF invoice that is sent as well the Inv. # is INV20161007
This is making my client very unhappy.
Can you help? Thanks