I am seeking advice regarding my ebook collection on a Linux system, which is stored on an external drive and sorted into categories. However, there are still many unsorted ebooks. I have tried using Calibre for organization, but it creates duplicate files during import on my main drive where I don’t want to keep any media. I would like to:
- Use Calibre’s automatic organization (tags, etc.) without duplicating files
- Maintain my existing folder structure while using Calibre
- Automatically sort the remaining ebooks into my existing categories/folder structure
I am considering the use of symlinks to maintain the existing folder structure if there is a simple way to automate the process due to my very large collection.
Regarding automatic sorting by category, I am looking for a solution that doesn’t require manual organization or a significant time investment. I’m wondering if there’s a way to extract metadata based on file hashes or any other method that doesn’t involve manual work. Most of the files should have title and author metadata, but some won’t.
Has anyone encountered a similar problem and found a solution? I would appreciate any suggestions for tools, scripts, or workflows that might help. Thank you in advance for any advice!
If the files are literally duplicated (exact same bytes in the files, so matching md5sums) then maybe you could just delete the duplicates and maybe replace them with links.
Automatically sorting books by category isn’t so easy. Is the metadata any good? Are there categories already? ISBN’s? Even titles and authors? It starts to be kind of a project but you could possibly import MARC records (library metadata) which have some of thatinfo in them, if you can match up the books to library records. I expect that the openlibrary.org API still works but I haven’t used it in ages.
deleted by creator
I tried to ingest a four terabyte epub library once. Even getting the data ingested with the author and title in the right spot was almost impossible. If a duplicates weren’t just slightly wrong would be a different story but the duplicates are often misspells or different spellings.
Realistically the best thing you can do is get an output of file name, title, author and hand dedupe, but even then you’re going to have to be careful about quality and language and all kinds of other strange issues you run into with large libraries.
In the end I gave up and only stored what I really wanted and would realistically ever need and that was small enough to hand cull.
I’ve run into exactly the same issue with my large ttrpg ebook/pdf collection (+100k file data hoarding… it’s not a problem, I swear) and I’ve not really found a good option I’m entirely happy with. Calibre duplicates everything and I don’t like the thought of having my collection’s organization tied to a specific piece of software if I just delete my duplicates. Plus I’m elitist and think the UI/logo are gross to look at.
Zotero is the least worst option I’ve found, but it’s geared towards scholarly journals and such, so not great, but serviceable. Not sure if it’s on linux though.
Jellyfin is apparently able to handle ebooks with a plugin, though I didn’t particularly care for it when I tried it months ago.
There’s a handful of other ebook software out there, mostly geared towards comics/manga, so depending on what you have those might be worth looking for.
I’d like to use Obsidian for it and just turn the directory into a vault and let it automatically scan the folders for files, but that doesn’t work great either.
The best piece of software I’ve seen that could potentially handle it is an app called Stashapp… which is unfortunately geared towards adult film. But it’s feature-set if it could be applied to PDFs seems like it would be ideal.
Why is duplicate files a problem?
I don’t like keeping duplicate files, especially in my main drive where I don’t store media. If I didn’t mind duplicate files, it wouldn’t be an issue.
I guess I’m a little confused… why are there duplicates at all? What operation are you performing that ends with duplicates?
Is this drive just where you download them to, and then move them to your organized drive? Why do the books ever touch this drive at all? If there isn’t supposed to be media on the drive, why not just delete the source folder after the organization task is complete?
Distilbert https://huggingface.co/distilbert/distilroberta-base
…was setup for something like that here but note that the repo that runs this has an “unsafe” warning that I have not looked into: https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier
https://huggingface.co/spaces/nasrin2023ripa/multilabel-book-genre-classifier/tree/main
It might be fine or whatnot, I’m on mobile and can’t see the file in question. The associated Python code might be a helpful starting point.
In my experience, most models intentionally obfuscate copyright sources. They all know the materials to various degrees, but they are not intended to replicate sources. They all have strong interference in place to obscure both their recognition and reproduction potential. If, for instance, you can identify where errors are inserted and make a few corrections, they often continue adding a few details that are from the original source. If this is done a few times in a row, they tend to gain more freedom before reverting to obfuscation again. This is the behavior I look out for. It is a strong tool too if you get creative in application.
Perhaps someone posts an API to look up the library of congress classification of a work based on a few lines or something. GL
In regards to another reply, there are multiple options of dealing with duplicates in calibre. From merging to deleting them.
It’s been a while since I had to do any library setup with calibre and Im kinda confused by what you are saying with “but it duplicates files when importing in a different drive where I don’t want ebooks”
Do you have multiple libraries in calibre?
Are you using multiple library managers?
Why not just use calibres folder structure?
And also if you don’t know about it - mobileread forums has been around a long time and has a whole section on calibre where you could probably get more specific help.
