Skip to content

About

Asynchronous document processing and vector embedding pipeline built with .NET 8 Isolated Azure Functions and Azure Service Bus.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Repository files navigation

Document Processor Function

This is a small practice project I built to try out Azure Functions with Azure Service Bus. The idea is simple: one app sends a message saying "hey, a document needs processing", and another app (the Azure Function) picks it up and processes it.

Right now the actual processing part (generating embeddings, saving to a database) is just faked with a delay and some log messages, since the point of this project was to get the messaging pipeline working first.

What's in here

There are two projects in the solution:

  • DocumentProcessorFunction – the Azure Function. It sits and waits for messages on a Service Bus queue called document-queue. When a message shows up, it "processes" it.
  • DocumentProducer – just a tiny console app. All it does is send one test message to the same queue so you have something to trigger the function with.

How it works (basically)

  1. You run DocumentProducer, and it sends a message like { "documentId": "...", "fileName": "..." } to the queue.
  2. The function wakes up because it's listening to that queue (ServiceBusTrigger).
  3. It logs that it got the message, then pretends to:
    • Generate vector embeddings for the document
    • Save those embeddings somewhere (like Cosmos DB or Postgres)
  4. That's it — job "done" (log message says so).

One thing worth knowing: Service Bus can sometimes deliver the same message more than once, so if you ever replace the fake logic with real logic, make sure running it twice for the same document doesn't cause problems.

What you'll need before running this

  • .NET SDK (8 or later)
  • Azure Functions Core Tools, so you can run the function locally with func start
  • An actual Azure Service Bus namespace with a queue named document-queue
  • Azurite or the Storage Emulator running locally (the function needs some storage to work, even if it's not the main point)

Setting up connection strings

Both projects need to know how to connect to your Service Bus.

For the function, that goes in local.settings.json:

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet-isolated",
    "ServiceBusConnection": "<put your connection string here>"
  }
}

For the producer, just edit the connectionString variable at the top of DocumentProducer/Program.cs.

Heads up: don't commit real connection strings to GitHub. It's fine for quick testing on your own machine, but if this repo is public, swap in a placeholder before pushing, or better, use user secrets / environment variables instead.

Running it yourself

  1. Make sure Azurite is running (or point storage settings at a real account).
  2. Start the function:
    cd DocumentProcessorFunction
    func start
  3. Open another terminal and run the producer to fire off a test message:
    cd DocumentProducer
    dotnet run
  4. Switch back to the function's terminal — you should see it pick up the message and log the (fake) processing steps.

Folder layout

DocumentProcessorFunction.slnx
├── DocumentProcessorFunction/   -> the Azure Function itself
│   ├── ProcessDocumentFunction.cs
│   ├── Program.cs
│   ├── host.json
│   └── local.settings.json
└── DocumentProducer/            -> sends a test message
    └── Program.cs

Things I still want to add

  • Parse the message into a proper class instead of a raw string
  • Actually call an embedding model (probably Azure OpenAI)
  • Actually save the results somewhere real
  • Get secrets out of the code and into somewhere safer

About

Asynchronous document processing and vector embedding pipeline built with .NET 8 Isolated Azure Functions and Azure Service Bus.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages