pg_durable: Microsoft open sources in-database durable execution (github.com)

Yeah, it's harder to work on, or maybe just different, but I guess the docs, info(searchable docs, posts, experience), and tooling are lacking.

What's the story for version control, debugging, testing, releasing? It'd be cool to have everything together for data locality and simplifying the stack, but it feels you'd lose a lot of useful knowledge about how to do stuff "properly".

+1 on "prefer my queue logic to be in code". The <shape> of my data doesn't change nearly as much as the actions I need to take on it; it doesn't make sense to me why I'd want to do a migration (which is an all or nothing op btw) every time I want to change how I behave with my data. This is also why I absolutely abhorred having to make postgres functions to do anything remotely non-trivial on Supabase.

That said, we did hand-build a simple job queue (just lock, poll, reserve on a column, poll and update reservation to mark job done) on top of postgres at my previous startup. Something like pgque would have made that much more polished.

same but this could be useful for db level things that are not business logic related.

i have always had maintenance packages for this type of stuff. if i could deploy them alongside the database itself that could be kind of cool.

but yeah i agree with you that i do prefer having this in the code layer.

I aggree - I'm not understanding the value of the project either if you look at the example here https://github.com/microsoft/pg_durable/blob/main/examples/i...

It's an interesting technical achievement I guess, but it's very bizarre to try and read this

    SELECT df.start(
        @> (
            ($$SELECT ... FROM demo.invoices WHERE status = 'pending'$$ |=> 'inv')
            ~> df.if_rows('inv',
                $$UPDATE ... SET status = 'processing'$$
                ~> (df.http(...) |=> 'resp')
                ~> df.if($$SELECT $r.ok$$,
                    -- classify, branch, wait for signal ...
                ),
                df.sleep(5)
            )
        ),
        'invoice-approval-pipeline'
    );

Microsoft has their own Durable Task framewor[1] for that kind of stuff, and it supports both running as a self-hosted standalone service like temporal, and running serverless on Azure Functions. It actually predated airflow, temporal, etc., IIRC.

This one seems to be more database-specific use case. The advantage is probably that you can track the exact state of the job in the database itself, rather than having to cross-reference the workflow log with the codebase and trace through it line by line to figure out what the state is. Plus I assume it's less overhead and latency, and operationally one less thing to spin up.

[1] https://learn.microsoft.com/en-us/azure/durable-task/common/...

Long-running jobs on Postgres are not new at all. See pg_cron for one example. At the end of the day, these workloads would be running anyway against the database, whether triggered by an external component. HTTP queries from the database have also become more popular to avoid round-trips and failure points from additional components in data or AI pipelines. But yes, whether to bring the compute to the data or vice-versa is a design choice that has a lot of contention.

Gotta set up the fall to rake in the dough later with consulting fees

Snapshot PITR of your database means everything restores including the durable jobs at the PIT.

Don't need to synchronize the backups with anything else that is part of the same data store, good for ETL pipelines and other state machine type jobs.

If your ETL is mostly SQL anyway, then having the actual job being run on the same server helps as well.

It’s sometimes convenient if database is the only ”stateful” component in architecture.

Also if all the "state" is in one database, then you have better chance of getting consistent backups.

Contributor here. At Microsoft, our Postgres customers seem to split pretty evenly into 2 camps, those that want to do as much as they can in the database, and those that agree with your take - want to keep apps and compute outside the DB.

You can have well-integrated applicative workflows (eg: progress report on a permalink in your front end app), app-restart-proof resumable workflows, and it avoids adding an extra piece of infrastructure.

We use Postgres for that on https://transport.data.gouv.fr (Elixir app which does a fair bit of processing), and it helps.

Not familiar yet with pg_durable though, but I have used or implemented similar solutions and can relate.

Please see https://learn.microsoft.com/en-us/azure/horizondb/ai/ai-pipe...

(pg_durable committer here)

The provider is an extensibility point. We just shipped the simplest version of it. Happy to take contribs if someone sends a pgmq based provider!