You Don't Need a Separate Database for Durable Workflows

DBOS recently argued that Postgres is all you need for durable execution. However, a leading tech expert at Obelisk, [Name Not Mentioned], thinks that SQLite is a more suitable choice for managing workflow state. According to [Name Not Mentioned], many durable systems don't require a separate orchestration tier and that SQLite can handle transactional durable state without introducing additional complexity.

For many AI agents and experimentation workflows, a local SQLite database is all you need. It's a natural fit for Obelisk, where workflow progress lives in an execution log, and workflows replay from persisted history. Activities can be retried, and what matters most is keeping the workflow state safe and easy to inspect.

SQLite is appealing because it eliminates the need for a separate database service. There's no network hop, no extra control plane, and no new operational surface area just to keep workflow progress safe. Many systems, especially those with AI-generated workflows, benefit from using SQLite as their database of choice.

One of the primary concerns with using SQLite is what to do with the database files once experiments start to accumulate. This issue is addressed by Litestream – a tool that can stream SQLite changes asynchronously to S3-compatible object storage. This gives you a simple way to keep working state close to the runtime while still backing up databases for inspection and analysis.

SQLite is not the answer to every deployment shape, however. Obelisk also supports Postgres, which is a better fit when you need higher availability, shared scalability, or other deployment properties that are better served by a network database. [Name Not Mentioned] agrees that Postgres is a good choice in these situations and that SQLite should be considered only when durability is the primary concern.

'SQLite is a natural fit for Obelisk, where workflow progress lives in an execution log, and workflows replay from persisted history.'

### The Benefits of SQLite for Durable Systems

The benefits of using SQLite for durable systems are obvious. Firstly, it eliminates the need for a separate database service, which reduces complexity and increases simplicity. Secondly, it provides a natural fit for Obelisk, allowing for easy management of workflow state. SQLite also offers a cost-effective option for many systems, reducing overhead and increasing efficiency, thereby making it an attractive choice for AI agents and experimentation workflows.

### The Future of Durable Systems

As more experts realize the benefits of using SQLite, we can expect a shift towards this database management system. Whether it's for AI agents, experimentation workflows, or other types of durable systems, SQLite is an option to consider. It's likely that SQLite will continue to grow in popularity, driven by its simplicity, cost-effectiveness, and ease of use.

### Key Facts

  • SQLite is a database management system that can handle transactional durable state without introducing additional complexity.
  • Litestream is a tool that can stream SQLite changes asynchronously to S3-compatible object storage.
  • SQLite is a natural fit for Obelisk, allowing for easy management of workflow state.
  • Postgres is a better fit when you need higher availability, shared scalability, or other deployment properties that are better served by a network database.
  • SQLite is a cost-effective option for many systems, reducing overhead and increasing efficiency.