Can't access documents after frontend upload

When a user on my site assigned as registered and managers uploads documents, the documents are not accessible. When going to download as a public user we get the message "You are not allowed to perform this action." When I upload the document from the frontend as a super user, the downloading of the document works fine for public users.