Machine Learning

1 readers

1 users here now

Community Rules:

Be nice. No offensive behavior, insults or attacks: we encourage a diverse community in which members feel safe and have a voice.
Make your post clear and comprehensive: posts that lack insight or effort will be removed. (ex: questions which are easily googled)
Beginner or career related questions go elsewhere. This community is focused in discussion of research and new projects that advance the state-of-the-art.
Limit self-promotion. Comments and posts should be first and foremost about topics of interest to ML observers and practitioners. Limited self-promotion is tolerated, but the sub is not here as merely a source for free advertisement. Such posts will be removed at the discretion of the mods.

founded 11 months ago

MODERATORS

communick@academy.garden

[D] Made some promises. Time to learn how to conduct very large scale pdf doc analysis. (alien.top)

submitted 10 months ago by -rampant@alien.top to c/machinelearning@academy.garden

4 comments fedilink hide all child comments

I have about a half million pdfs I need to summarize. Very wide range of types: invoices, diagrams, contracts, emails, letters, pictures, schedules, notices, data sheets, manuals, more.

Which is... woof. Something else. I've been trying for many hours now to figure out a service/combination thereof that can get me there, but I'm seriously struggling. The ideal solution would be to throw the pdfs in and have it return a csv with dates and summaries, maybe parsed out email heading info.

I'm currently running these pdfs through Acrobat OCR now, which its own special hell.

I've tried myriad local and webhosted solutions. The BEST results in what is almost the perfect system for this I found on https://docalysis.com/. Good text results, works in batches, BUT I can only upload a single document at a time. They have a service to do batch processing and so I'm waiting to hear from them now. I imagine at the scale I need it's expensive.

I also got this solution working: https://github.com/mayooear/gpt4-pdf-chatbot-langchain. Seemed solid, I was able to upload a thousand pdfs in a single go, but it would keep returning information from only 2-3 documents. Upload 5? Results for 2-3. Upload a thousand? Results for 2-3. My uneducated guess is that it's hitting the OpenAI API token limit, but maybe not?

I know it's possible, just not whether it's feasible for an end user.

you are viewing a single comment's thread
view the rest of the comments

[–] Zahlii@alien.top 1 points 10 months ago

If it’s just about classifying documents nothing stops you from iteratively hand labeling a new set of documents, training a model to suggest classes, correcting the model predictions on a new set of documents and repeat. Will be cheaper than using complex models and may get you there. If it’s about making the whole stuff indexable best bet is to use a vector database, sentence transformer embeddings and then putting the top 4-5 closest paragraphs to your search into a LLM prompt for reranking. If it’s about extracting structure information from each document you will most likely need to use a) specialized custom model or b) one LLM prompt for each document