Dotch.app: Testing

Dotch works! Here’s a video — still buggy, but it works.

You can try it out yourself at https://dotch.app. If you have any feedback, please send to dotch.app@gmail.com.

Technical Design

I built the frontend with Vue 3 and the backend with Node JS. The API has two routes: POST /scan and GET /receipt/:id.

It all starts from the frontend. A WebCam component takes a photo using HTML5 navigator and prints that image onto a canvas.

The frontend sends the image to the server as a Base64 file, which is then formatted and passed to the Mindee Node JS client for parsing.

const getDocument = async (request) => {
  let base64String;
  const parts = request.parts();
  for await (const part of parts) {
    if (part.value) base64String = part.value;
  }
  if (base64String.includes(";base64,")) {
    base64String = base64String.split(";base64,")[1];
  }
  try {
    const inputSource = mindeeClient.docFromBase64(
      base64String,
      "receipt-data",
    );
    const apiRequest = mindeeClient.parse(
      mindee.product.ReceiptV5,
      inputSource,
    );
    const apiResponse = await apiRequest;
    const { document } = apiResponse;
    return document.inference;
  } catch {
    return null;
  }
};

Mindee returns the document type, labels and line items. I use this to decide if the document is a receipt. If it is, I filter out the important information and send it back to the client to display.

const isReceipt = (document) => {
  const { documentType, lineItems } = document.prediction;
  const docIsReceipt = documentType.value.toLowerCase().includes("receipt");
  const hasLineItems = lineItems.length > 0;
  const docIsConfident = documentType.confidence > 0.6;
  return docIsReceipt && hasLineItems && docIsConfident;
};

const filterReceiptFields = (prediction) => ({
  merchant: prediction.supplierName.value,
  date: prediction.date.value,
  currency: prediction.locale.currency,
  items: prediction.lineItems,
  taxes: prediction.taxes,
  tax: prediction.totalTax.value,
  total: prediction.totalAmount.value,
});

Notes

• There’s no database yet. I’m hoping to implement lowdb in the future but for now I use a variable to store uploaded receipts. This means every time the server is restarted, all receipts are refreshed. You can see the code on Glitch here: https://glitch.com/edit/#!/dotch
• The line items are often wrong. For best results, put the receipt on a flat surface and clean your camera. I need to improve the isReceipt logic to aggregate the confidence score per line item and if it’s too low, request a better image from the client.
• ChatGPT and Copilot were such a great help! I still used Google and Stack Overflow for specific issues, but sparingly. Here’s a log of all the requests I asked ChatGPT for: https://chat.openai.com/c/071b04c2-553f-4a64-af1b-b74b95020ff4 What next?

I’m testing, getting feedback and improving the app in my spare time. You can try it out yourself at https://dotch.app. If you have any feedback, please send to dotch.app@gmail.com.