ChatDOC Review – Critical look at an AI PDF Tool

In this post, we conduct a ChatDOC review – an online AI tool that allows you to “speak to your documents” via a tailored interface that is powered by ChatGPT. At Vitr Tech, we keep a close eye on a number of promising software tools which may help evolve how businesses operate. Up front, most ‘chat-with-your-document’ AI-based Chat Bots under-perform in a range of use cases and this tool is no exception. However, the limitations are largely based on the existing limitations of the large language model (LLM) technologies in existence. As far as existing tools currently go, we’ve found this to be reasonably effective in certain scenarios.

As part of this review, we’ll primarily focus on ChatDOC Pro, but will briefly outline some of the key differences between the paid and free versions of the software. This ChatDOC review will also detail some of its key features, and also analyse the efficacy and benefit of adopting this tool for businesses involved in digital transformation. To help guide the review, we’ll routinely compare this tool to ChatGPT Plus (the paid version) which allows you to create your own GPTs with similar (albeit more limited) results.

Overall, this ChatDOC review rates the tool 3.5 out of 5 stars. Our rating reflects the tools useful features when compared to ChatGPT Plus but incorporates the limitations found in most ‘chat-with-your-document’ AI tools found on the internet.

The AI Tool – ChatDOC

ChatDOC presents itself as an enhanced ChatGPT based tool used for managing and synthesising complex document suites, beyond the current standards of existing LLM based Chat Bots. The free version of the tool is powered by ChatGPT 3.5 and for paying customers, ChatGPT 4 can be accessed. At its core, ChatDOC is designed to scan, analyse and summarise documents, providing a user-friendly interface that provides citation backed responses and the capacity to search, chat and interact with multiple documents at once. With support for a wide range of document formats—.pdf, .docx, .doc, .txt, .epub, URL, html, website, and scanned files—ChatDOC markets itself as a versatile tool for various types of users, especially businesses, professionals and academics.

Picture 1: ChatDOC Interface

Pricing & Feature

From the ChatDOC website, here is a comparison of features available between the free and pro versions.

Picture 2: ChatDOC Pricing

The pro version will set you back $8.99 USD ($17.00 AUD) per month and represents reasonable value for the capability you’ll be able to access. In comparison, ChatGPT 4 with its ‘mini-GPTs’ and integrated knowledge bases will cost you $20.00 USD ($31.00 AUD) per month and the databases are limited to 20 files per GPT with associated size restrictions. As highlighted in the above table, a key feature of ChatDOC is that it has no file limits and its ‘memory’ (or the length of conversation the Chat Bot can access) is longer than GPT 4 (more on this point below).

The unlimited file and memory capability of ChatDOC is really at the core of what makes ChatDOC an appealing tool for certain users. Looking at OpenAI’s ChatGPT 4, the API currently allows access to 128K tokens, effectively providing ChatGPT a memory of no more than a 300 page book in one query.  ChatDOC in comparison claims that their ‘retrieval augmentation’ system can support Q&A for documents up to 3000 pages. This is a significant improvement, particularly for businesses that are seeking to build an effective repository of commercial information that can be accessed via a chatbot.

When compared to ChatGPT, we actually believe the truly greatest feature of ChatDOC is its internal reference system. For every response it provides, it will also provide you a handy reference link that can allow you to immediately identify where in the document it is pulling its response. This is particularly handy due to the proliferation of hallucinations that often occur with most LLM chat bots.

Ultimately, if you’re looking to make a decision between paid versions of ChatGPT and ChatDOC, I think your decision will be based on the number of documents and their length, in addition to the specific outputs you’re  looking for. Price will also be a factor. A more expensive ChatGPT Plus subscription will provide you access to a more advanced LLM but you will face limitations regarding file size and number. Further, you will miss some of the key features that ChatDOC provides.

Privacy

One of the primary concerns around AI tools in business is privacy. We know that AI can be ‘hacked’ (read about prompt injections here) to retrieve information that should remain private.  These ‘hacks’ are being mitigated all the time, but for obvious reasons, remain a key concern, particularly with business proprietary information. As we’ve already touched on, ChatDOC uses ChatGPT in the back end, so if you’re using anything less than GPT Teams or Enterprise, which in this case it is, it’s important to be aware that your data is likely available to OpenAI to train their models.

This article provides a great overview of how ChatGPT (OpenAI) use your data. We know that ChatGPT recently allows users to opt out of training their models, but we were unable to access this feature through ChatDOC itself. Also, our research into ChatDOC didn’t uncover any specific comments or detail confirming ChatGPT training on your data, however they do however mention data storage. Read the comments about data storage on the ChatDOC website:

Ultimately, we still urge caution around particularly sensitive files.

User Experience.

One of the hallmark features of ChatDOC is its chat bot style interface which enable users to engage in conversations about the content of their documents. Most importantly, you can upload multiple documents to the drive and use the tool to compare, contrast and summarise multiple documents. We found that this still had an effective limit when we attempted to upload over 50 documents with hundreds of pages per document. Saying that, ChatDOC proclaims that it has a particular edge over other competitive models when it comes to PDFs.

As evidenced by this blog post on their website, ChatDOC was clearly challenged by the introduction of GPTs via ChatGPT 4. Yet, ChatDOC highlights the PDF parser used by ChatGPT (PyPDF2) extracts documents without formatting. They contrast this with the PDF parser developed by the ChatDOC team that allows PDF structure to be maintained by the AI, and can therefore be used to contextualise the data. This feature is particularly relevant for documents like financial reports or manuals that feature dense tables and multi-column layouts. Without this context, the relevant information can become lost or difficult to find by the Chat Bot.

In our testing of the software, we found that ChatDOC did indeed perform better on these types of format heavy PDF’s. To compare ChatDOC with ChatGPT, we uploaded a large (550 page) spare-parts manual into the two tools to analyse their performance in identifying a single part number, and providing a description of the part, including context around its usage.  We were analysing the tools usefulness when utilised in a parts-interpretation role or for a spare-parts sales process.

The PDF used was heavy in formatting and tables. To begin, it took some prompting with both systems to correctly identify the part. However, once we had successfully cued the system, we found ChatDOC’s responses to be more analytic, and they provided specific references to our document, which it also provided in a pane view to the left of the screen. In contrast, ChatGPT provided more general context about what the part could be for, but due to a lack of referencing, it was difficult to fact check the response against the source document.

Ultimately, we preferred ChatDOCs interface and for the role we were testing it against, it provided more useful responses. However, while it took ChatGPT 4 longer to correctly identify our part, once it did, its description of the part was more useful.

Picture 3: ChatDOC Testing

Picture 4: ChatGPT Testing

Perhaps our use case was a challenging one for the technology. The document we tested was dense in formatting and long by prompt standards. In another test we came across online, ChatDOC demonstrated its capability to provide concise summaries and answer specific queries related to the document’s content. This test utilised an IEEE research article that had fewer formatting challenges, but had a lot of dense information. This test found that ChatDOC was able to provide a very detailed summary of the document and could direct a user to key pieces of information in the source document via its internal reference system. We think this feature would be very useful for a business involved in knowledge management such as law firms and media.

Conclusion

When it comes to supporting a document repository for a small business, ChatDOC Pro’s tools appear to useful in certain circumstances. The ability to upload a significant number of documents and query them via an interactive chat feature can streamline the handling of extensive document collections. This capability is especially beneficial for businesses that need to manage, summarise, and query a large volume of documents regularly.

Noting some of the challenges around data privacy, and the fact your specific use case may not be well catered for, the tool is still worth testing and in my opinion will only improve as the underlying technology improves. There is an ever expanding suite of tools that leverage generative AI via innovative interfaces, and if you have a specific use case, get in touch with Vitr Tech and we can provide you specific advice on how to achieve it. We also offer a free digital transformation audit to identify the most impactful digital changes you can make your business.