Rendered at 18:36:16 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
k9294 22 hours ago [-]
I'm a big fan of SQLite embedded nature, which allows for chaining multiple SQL calls with near-zero latency.
I'm currently building a personal knowledge graph server a mix of Notion's custom entities via JSON schema and Obsidian markdown+backlinked references. It's working well, but I suspect your product might be a better fit.
I do have one question regarding permissions: how would you recommend modeling a hierarchical access system in a graph database? Specifically, if a user is granted access to a document, they should automatically have access to all its child documents within that workspace. Is there a standard way to model this 'subtree' permission logic, or perhaps a more efficient approach you'd suggest?
Really impressed with the product good luck with it!
infogulch 16 hours ago [-]
Modern authorization systems are often graph-based. Check out ReBAC and ABAC authorization models and also implementations like apache/casbin or authzed/spicedb. These schemas often have surprisingly simple graph definitions; I bet they could be replicated in LatticeDB without much trouble.
smiths1999 20 hours ago [-]
The permissions question is interesting. I think the answer depends on context. One approach would be to create some edge types `hasAccessTo` and `accessibleBy` that connect a user to a node. Then I'd create an edge type `childOf`. The rest is business logic. Permission checks can just traverse up to the first root node with permissions. The downside is this is all business logic, so can't really look at the database and understand this is how it works. Depending on the database you could create a function that returns permissions for any node, that encapsulates this logic.
Anyways, thanks for checking it out! Really appreciate it. Good luck with your project!
Very different from the comparison on github and the website.
Given that on-disk data structures are similar to SQLite, I expect the competition from other "graph on sqlite" projects when they co-opt the techniques in LatticeDB.
smiths1999 6 hours ago [-]
Thanks for sharing this. The lattice numbers are pretty close but my sqlite numbers are way off. I got a new computer since then so will remeasure.
anigbrowl 12 hours ago [-]
Very nice - been looking for something that combined the flexibility of graph DBs like Neo4j with the simplicity and low cognitive overhead of SQLite. PErformance looks great too.
Excellent Readme - as well as the clear explanations and examples, very impressed by the 'when you should use it/when you should NOT' use it' section - sadly often missing on many worthy but unfriendly tools.
smiths1999 6 hours ago [-]
Thanks - appreciate the feedback! I wanted the README to be something you could quickly glance at, get a sense of the project and whether it's what you are looking for, and be done with it.
tescreal 23 hours ago [-]
I see "Claude" listed as a contributor. Could you describe how and how much? I'm keen to see how this looks in practise.
As for the tool, it scratches an itch I've been having, I'll give it a go soon.
smiths1999 23 hours ago [-]
I used claude extensively (as well as codex, I finding myself switching between the two every few months). How I used claude varied by the stage. I spent a lot of time initially going back and forth with claude on the idea, figuring out what exists, what would make this interesting, key features I wanted as a user and how to build something around that that made sense as a product.
The initial phases of building I would build out piece by piece. For example, building out the file system interactions I would have claude build a feature and explain how it worked in an educational manner (e.g., like it was a section in a book on latticedb internals). I would then read through the code. This was a great way to learn and build, simultaneously.
In the later stages, where the features and work was more complex, I would spend more time discussing, instructing, and verifying, but less time understanding the actual implementation. I'll give you an example. It's been a long time since I've handwritten SIMD code. I could try and review claudes output, but I am certain I'd miss any subtle bugs that may exist. I found it more productive to assume the code was right and focus on thinking about how I would verify that. Benchmarking, playing with latticedb, etc. were my primary tools for verifying the work. I could run a benchmark and see performance was great. Then I'd explore the test vectors and realize they were trivial, completely invalidating the benchmark results. So we would go back to the drawing board, create a new benchmark set, see results weren't great, and evaluate what was wrong with the implementation. Sometimes features would take days to get out just because of the iteration loop.
pwmglenn 3 hours ago [-]
How does it compare to embedded SurrealDB?
smiths1999 2 hours ago [-]
Not comparable. Similar to Neo4j vs LatticeDB. Lattice's goals are simplicity, single file, super easy to start using. Surreal supports embedded mode, but my understanding is it is not a single file. Surreal supports a huge number of data models, Lattice is more focused. Surreal also appears to have a company behind it. Lattice is just me.
If you are looking for something lightweight and simple, choose Lattice. If you are an Enterprise looking for hosting, Enterprise support, and maybe a database that can handle all of your needs instead of a specific one, pick SurrealDB.
I hadn't heard of SurrealDB until your comment though, so take what I have to say here with a grain of salt.
cjlm 22 hours ago [-]
Nice, I’ll get it added to gdb-engines.com
EGreg 17 hours ago [-]
Thanks, that's a great list! Might be worth mentioning our engine there too, for people who want a graph database atop SQLite and other battle-tested relational databases like MySQL, MariaDB, Postgres.
Our database abstraction layer (and optional ORM) has been battle tested in production for millions of users, and has 3 adapters: Sqlite, Postgres, MySQL/MariaDB. And recently it even added vector search for ranking results by similarity: https://github.com/Qbix/Platform/tree/main/platform/classes/...
PS: If you do use a relational database for storing graph data, you're going to have a lot of duplication in some public keys. I highly recommend putting ZFS underneath, to help with deduplication. ZFS uses zstd, developed at facebook, and also can encrypt your data at rest (don't use the relational database to do the encryption, otherwise deduplication doesn't work).
smiths1999 21 hours ago [-]
very cool site!
itissid 22 hours ago [-]
Does it have something like litestream to backit up for specific production usecases (i.e. a single webserver is enough and downtime of a few mins is tolerable)?
smiths1999 20 hours ago [-]
I am in the final stages of adding this based on your comment. Just wrapping up doc updates and will push a new release with hot copy functionality tonight!
moostee 11 hours ago [-]
champion
petervandijck 23 hours ago [-]
Congrats this is really cool and love the examples
tomComb 23 hours ago [-]
I wonder about mapping RDF data (like Wikidata) to this. I guess the RDF predicate becomes the edge in your node-edge style of graph.
adsharma 16 hours ago [-]
Wikidata doesn't imply RDF/SPARQL. Cypher works fine too. A columnar storage engine means you get indexes and the relational goodness for free.
Yes! This is something I've been thinking about quite a bit the past few weeks. We are going in this direction at work and I think graph storage is a natural way to think about this.
LunaSea 9 hours ago [-]
Interesting, I used to work quite a lot with triples from ontologies.
Since the release of LLMs, do you still see use cases for ontologies and triples (RDF, OWL or otherwise) compared to simply feeding it to an LLM?
zvr 22 hours ago [-]
If you're going to support RDF, please also consider supporting SPARQL for querying the data.
ebolyen 9 hours ago [-]
For my own edification, what is the goal of SPARQL compared to normal SQL or if you live in predicate-land, Prolog or Datalog?
The semantic web tools never seemed to have their breakout moment and seem to run in parallel to "mainstream" history of relational and logic programming, but I've never really understood why.
smiths1999 20 hours ago [-]
I started thinking more deeply about how I would actually do this today and it won't be as straight forward as I initially thought. Will keep it on the radar and see if I can figure out a clean way to implement. Great suggestions!
srameshc 1 days ago [-]
LatticeDB looks good, just curious how useful is duckpgq
duckpgq is great. I'd say the tl;dr is duckpgq if you have tables you want to traverse like a graph sometimes, latticedb when graph traversal is the primary mechanism of querying.
One of the motivating use cases for me was experimenting with agentic memory. I use latticedb as the backing data store. Finding related memories is traversing the graph (kind of like graph RAG).
vladigtr 22 hours ago [-]
Nice work. How do you handle concurrent writers on a single file? That's where embedded databases usually get tricky, and the graph model makes locking even more interesting.
smiths1999 20 hours ago [-]
Within the same process it's enforced at the db level. After reviewing the code more I see we don't have a mechanism in place to manage safety across processes. Will add a file lock mechanism tonight. Great call out!
In general though, the goal for this was single writer multiple readers. That was a design decision to keep things simple.
smiths1999 5 hours ago [-]
Just added file locking to avoid concurrent writers across processes in the latest release.
ebb_earl_co 1 days ago [-]
Just read through the README on GitHub and this looks impressive! Kudos
smiths1999 23 hours ago [-]
Thank you!
vorpalhex 1 days ago [-]
Thank you for sharing. I think the sqlite-esque local file approach makes sense for a lot of use cases.
What are some of the scales of the data you've been able to test this design on so far?
What was the most interesting part of designing it for you?
smiths1999 24 hours ago [-]
I did some perf benchmarking with 1M nodes but I'm mostly using it at smaller scales for another project exploring agentic memory.
Most interesting part is a tough one. From a learning perspective the beginning was incredibly interesting because I was spending a lot of time learning about how other DBs work. Even something as relatively simple as writing to disk had a lot more complexity to it than I initially anticipated.
I used LLMs extensively in building this, and the other interesting part was seeing how they failed. I've always been a proponent that tests are no guarantee of quality code, and working with LLMs has only reinforced it. They often write superficial tests. Sometimes a suite of tests would pass, but when I would actually play around with the feature it was clearly broken. LLMs certainly enabled me to build something of this scope, but it was far from "build a graph DB and notify me when you are done"
nrjames 22 hours ago [-]
Out of curiosity, why did you not fork and build on Kuzu?
smiths1999 22 hours ago [-]
Great question! Two reasons. One, I wanted to build something on my own from the ground up rather than contribute to an already established large project. For me, it's a better way to learn. Hopefully people find it cool and want to use it. But if the best that happens is it's just a fun project I built then that is just fine for me. Secondly, the data layouts are different. They are very similar in the single-file, graph db respect. But lattice is transactional and row oriented while kuzu is columnar.
* LadybugDB has revamped the Kuzu WAL design. It shouldn't be hard to build WAL based replication
* 19ms vs 39us - like the author says these are vastly different systems and the benchmark methodology may not be comparable.
We've mostly focused on query plan optimizations, not so much the micro query operator optimizations.
The 0.20.x end of the month release should have some interesting optimizations.
* Prepared statements will cache query plans and result vectors. So you don't pay malloc costs
* SIMD optimizations for filter. More to come in the next release.
lmeyerov 19 hours ago [-]
Congrats!
For those into the `pip install ...` flow and kuzu, is gfql: we started around the same time in a non-VC-funded oss manner with overlap in key architectural ideas:
- cpu columnar vectorized engine + optionally the only open source gpu engine mode for bigger graphs / faster queries
- removes the need for a database / file: pure compute-tier engine you can write to parquet/json if you want, plays with parallel reader/writers in simple ways b/c that, and TBD iceberg
- adds full graph analytic pipeline support, eg, for feature engineering in real-time fraud & memory pipelines
- also millisecond/submillisecond times on small graphs like that small 100K edge graph benchmark
Main box not formally checked is streaming. Funny enough, we're designed for GPU firehose workloads, so would be fun to demo and see what gaps are left.
I'm currently building a personal knowledge graph server a mix of Notion's custom entities via JSON schema and Obsidian markdown+backlinked references. It's working well, but I suspect your product might be a better fit.
I do have one question regarding permissions: how would you recommend modeling a hierarchical access system in a graph database? Specifically, if a user is granted access to a document, they should automatically have access to all its child documents within that workspace. Is there a standard way to model this 'subtree' permission logic, or perhaps a more efficient approach you'd suggest?
Really impressed with the product good luck with it!
Anyways, thanks for checking it out! Really appreciate it. Good luck with your project!
Given that on-disk data structures are similar to SQLite, I expect the competition from other "graph on sqlite" projects when they co-opt the techniques in LatticeDB.
Excellent Readme - as well as the clear explanations and examples, very impressed by the 'when you should use it/when you should NOT' use it' section - sadly often missing on many worthy but unfriendly tools.
As for the tool, it scratches an itch I've been having, I'll give it a go soon.
The initial phases of building I would build out piece by piece. For example, building out the file system interactions I would have claude build a feature and explain how it worked in an educational manner (e.g., like it was a section in a book on latticedb internals). I would then read through the code. This was a great way to learn and build, simultaneously.
In the later stages, where the features and work was more complex, I would spend more time discussing, instructing, and verifying, but less time understanding the actual implementation. I'll give you an example. It's been a long time since I've handwritten SIMD code. I could try and review claudes output, but I am certain I'd miss any subtle bugs that may exist. I found it more productive to assume the code was right and focus on thinking about how I would verify that. Benchmarking, playing with latticedb, etc. were my primary tools for verifying the work. I could run a benchmark and see performance was great. Then I'd explore the test vectors and realize they were trivial, completely invalidating the benchmark results. So we would go back to the drawing board, create a new benchmark set, see results weren't great, and evaluate what was wrong with the implementation. Sometimes features would take days to get out just because of the iteration loop.
If you are looking for something lightweight and simple, choose Lattice. If you are an Enterprise looking for hosting, Enterprise support, and maybe a database that can handle all of your needs instead of a specific one, pick SurrealDB.
I hadn't heard of SurrealDB until your comment though, so take what I have to say here with a grain of salt.
My team and I built it over the years, and it's open source (AGPL). Here is how it works: https://community.qbix.com/t/qbix-streams-as-a-graph-databas...
Our database abstraction layer (and optional ORM) has been battle tested in production for millions of users, and has 3 adapters: Sqlite, Postgres, MySQL/MariaDB. And recently it even added vector search for ranking results by similarity: https://github.com/Qbix/Platform/tree/main/platform/classes/...
Documentation for the database layer is here: https://qbix.com/platform/guide/database
PS: If you do use a relational database for storing graph data, you're going to have a lot of duplication in some public keys. I highly recommend putting ZFS underneath, to help with deduplication. ZFS uses zstd, developed at facebook, and also can encrypt your data at rest (don't use the relational database to do the encryption, otherwise deduplication doesn't work).
https://huggingface.co/datasets/ladybugdb/wikidata-20260401
Since the release of LLMs, do you still see use cases for ontologies and triples (RDF, OWL or otherwise) compared to simply feeding it to an LLM?
The semantic web tools never seemed to have their breakout moment and seem to run in parallel to "mainstream" history of relational and logic programming, but I've never really understood why.
https://duckdb.org/community_extensions/extensions/duckpgq
One of the motivating use cases for me was experimenting with agentic memory. I use latticedb as the backing data store. Finding related memories is traversing the graph (kind of like graph RAG).
In general though, the goal for this was single writer multiple readers. That was a design decision to keep things simple.
What are some of the scales of the data you've been able to test this design on so far?
What was the most interesting part of designing it for you?
Most interesting part is a tough one. From a learning perspective the beginning was incredibly interesting because I was spending a lot of time learning about how other DBs work. Even something as relatively simple as writing to disk had a lot more complexity to it than I initially anticipated.
I used LLMs extensively in building this, and the other interesting part was seeing how they failed. I've always been a proponent that tests are no guarantee of quality code, and working with LLMs has only reinforced it. They often write superficial tests. Sometimes a suite of tests would pass, but when I would actually play around with the feature it was clearly broken. LLMs certainly enabled me to build something of this scope, but it was far from "build a graph DB and notify me when you are done"
Couple of corrections:
* LadybugDB has revamped the Kuzu WAL design. It shouldn't be hard to build WAL based replication
* 19ms vs 39us - like the author says these are vastly different systems and the benchmark methodology may not be comparable.
We've mostly focused on query plan optimizations, not so much the micro query operator optimizations.
The 0.20.x end of the month release should have some interesting optimizations.
For those into the `pip install ...` flow and kuzu, is gfql: we started around the same time in a non-VC-funded oss manner with overlap in key architectural ideas:
- cpu columnar vectorized engine + optionally the only open source gpu engine mode for bigger graphs / faster queries
- removes the need for a database / file: pure compute-tier engine you can write to parquet/json if you want, plays with parallel reader/writers in simple ways b/c that, and TBD iceberg
- adds full graph analytic pipeline support, eg, for feature engineering in real-time fraud & memory pipelines
- also millisecond/submillisecond times on small graphs like that small 100K edge graph benchmark
Main box not formally checked is streaming. Funny enough, we're designed for GPU firehose workloads, so would be fun to demo and see what gaps are left.
However, I really miss the content posted by the KuzuDB team on their YouTube channel.