Fast Analytics and Graph Traversals with Datalog
Is it possible, or better ask how is that possible ? Although graph databases and knowledge bases are becoming more and more popular for managing and querying complex interconnected data one has to consider the potential trade-offs they present in terms of scalability, performance, and maintenance. Most important, rules specifications and application must be utilized for business logic, data validation and quality, security and access control, automation and inference among others.<p>In the last decade, one approach that has gained traction in addressing the challenges of graph data management is leveraging Datalog to cope with the complexity of graph data modeling, data transformations and intricate queries. Since it is a declarative logic programming language, it also offers the ability to express rules, relationships and constraints in a concise and intuitive manner. But it is only recently that appeared on the scene production systems that are based on *relational knowledge graph data models with analytics capabilities* powered by elevated versions of Datalog.<p>There are two of them that have drawn my attention, Relational AI and CozoDB. The former is a proprietary commercial product that runs on top of  Snowflake columnar DBMS and the later is an excellent open-source embed-able  production ready DBMS with several storage engines where I spent some time investigating and exploring its features. Bottom line is that these relational graph Datalog systems are covering a huge gap between SQL, noSQL, newSQL, GQL query languages at one side and Prolog&#x2F;Lisp logic programming languages at the other side. According to Ziyang Hu creator of CozoDB &quot;composability&quot; of Datalog is the main reason that drove him to adopt it as the query language and he is not alone.<p>In my opinion the main obstacle for a broad adoption of Datalog based systems in graph analytics  and traversals is their performance on big datasets. Is it possible to utilize vectorized parallel execution either in-memory with a fast dataframe structure like Polars or on the disk with an equally fast columnar DBMS like ClickHouse ? Truth is that I have not seen any Datalog system implementation with columnar data processing but I have discussed informally this with developers and they found that it is an interesting idea. In the past, I attempted to import a file of several hundred megabytes and execute queries on triple store Datalog systems such as Datomic, Datalevin, XTDB  and the performance was very disappointing. These are mostly transactional, not built for analytics and queries on big data. CozoDB performance is a lot better, but it is still several orders of magnitude behind columnar engines&#x2F;dataframes. The URL of my question points to a little benchmark I wrote to compare CozoDB with ClickHouse and Polars.<p>I am very interested in that technology and I am also trying to understand any technical issues that prevents someone from combining Datalog with a columnar database&#x2F;dataframe. Let me know what you think, I am also willing to contribute in such a project.