My computer science PhD thesis contains a chapter which is heavily based on an already published conference paper. I have published my thesis online, and fulfilled the requirements for my degree this way.
Nevertheless it is customary in my group to also publish the thesis as a proper book, to get shiny printed copies and to make it available to libraries etc.
Now I was invited to submit an extended version of my conference article to a journal, and since the thesis chapter was already an extended version of the conference article, I used material from there and the journal paper turned out to be very similar to the thesis chapter.
Will the journal publisher (Elsevier) likely accept a paper that has significant overlap with a chapter in a book that is not peer-reviewed, but (by then) is commercially published?
(This question is about copyright and publishing, not about citations and giving proper attribution to the other work.)
(Note: This answer doesn't apply to hard-copy theses provided to your university library as part of your degree requirements. These count as theses for the paragraph on exceptions.)
You're dealing with two separate issues here. The scholarly ethics of duplicate publication and the copyright issues. Let's tackle them one by one.
Scholarly Ethics
Most academic journals will not consider for publication work that his previously been published elsewhere (or is being considered for publication elsewhere); furthermore, the submitting author will usually be required to explicitly confirm that the work is not published (nor being considered for publication) during submission. This is known as the Ingelfinger rule. Subject to a few exceptions, discussed momentarily, the mode of publication doesn't matter: peer-reviewed paper, conference proceeding, book/book chapter, newspaper article, blog post, vanity publisher or predatory journal, it's been published. This doesn't simply apply to the manuscript, it applies to the work (So you can't rewrite the same analysis of the same data and claim the rule doesn't apply). You can read the reasons for (and against) this on Wikipedia.
Now, as I mentioned, there are some widespread exceptions, principally for things that happen in the normal course of academia. So inclusion in a thesis, even one that is made available online, is not usually considered publishing; similarly for conference presentations where no proceedings paper has been published. Expansion of communications into full papers is typically allowed. Increasingly, the use of pre-print servers is becoming permitted. Some journals/fields will permit submission of journal articles that heavily expand a conference proceeding, but in other fields this is a no-no.
Specific policies are set by individual journals/publishers. For example, Nature, Science and Elsevier.*
This means, when publishing, you should publish the peer-reviewed academic version first, then, subject to copyright, you can publish your book.
Minor consideration: (self) plagiarism. You should always reference the first publication of your work in any later re-publication, to avoid accusations of plagiarism (with a possible exception for publications derived from PhD theses). So even if a publisher allowed you to submit a journal article after you published your book, you would still have to cite your book.
*Interestingly, I did once inform Elsevier that a paper in one of their journals seemed to have been previously published elsewhere, and got ignored.
Copyright
Your options for republishing the paper as a book will vary depending on the copyright status of your publication. Here are some possible scenarios (not all will be available with all journals):
Open access publication that leaves copyright with the authors
- You can do what you want to republish the article, including republishing it as-is, since you hold the copyright
- You probably can't reuse the publisher PDFs for republication, as they will probably have trademarked logos on them
- You will need to appropriately reference your original paper to avoid self-plagiarism
Open access publication that leaves copyright with the journal
- You may be able to republish the paper under the open access license. This will depend on the license chosen by you/the journal.
- You will need to consider whether there are clauses restricting things such as commercial use and the preparation of derivative works
- You may find the license too restrictive, in which case proceed as for a non-open-access publication
Publication is not open access, copyright is with the journal
- You cannot reuse the article without permission from the journal
- You could try contacting the publisher, or using the Copyright Clearance Center to obtain permission
- Otherwise, you will need to ensure that your book is different enough from the publication that it is not an infringement of copyright
- Typically, this would require completely rewriting it from scratch (not merely paraphrasing), ideally restructuring it and re-drawing all the figures (trying to make them as different as reasonably possible from the journal version)
- You may be able to obtain permission for specific components of the paper (e.g. figures) individually via the Copyright Clearance Center. In that case, you could use them as is, with appropriate acknowledgements
- Academic ethics still dictates that you cite that your source is your original paper
Summary
Publish the paper first, then the book. You state that the book is merely "customary" - it can wait. Consider the copyright issues before publishing the book - this may mean you have to either seek permission from your publisher or modify your thesis chapter, if it is too close to the published paper. Protect yourself against allegations of self-plagiarism - ensure that your paper is very clearly listed as a source in your book.