SQLite-Based Binary Logging

MODULE.bazel

@@ -81,6 +81,7 @@ bazel_dep(name = "googletest", version = "1.17.0.bcr.2")
 bazel_dep(name = "openmp", version = "21.1.5.bcr.1")
 bazel_dep(name = "or-tools", version = "9.15")
 bazel_dep(name = "spdlog", version = "1.15.1")
+bazel_dep(name = "sqlite3", version = "3.51.2")
 bazel_dep(name = "sv-lang", version = "10.0.bcr.2")
 bazel_dep(name = "tcl_lang", version = "9.0.2.bcr.1")
 bazel_dep(name = "tcmalloc", version = "0.0.0-20250927-12f2552")

docs/contrib/Logger.md

@@ -454,3 +454,36 @@ cd doc/messages && touch <NUM>.md
 
 ### OpenROAD Tool List
 A full list of tool namespaces can be found [here](DeveloperGuide.md#tool-flow-namespace).
+
+# Binary/Database Logging
+There is also a parallel mechanism for logging large quantities of numeric data to an SQLite database, useful for analysis of the inner workings of tools. 
+
+## Overall Architecture
+The architecture is designed around many threads doing logging, with many different log sources as well. This requires buffering everything, with a background thread draining the buffers into the database. The scheduling is a simple round-robin in the common case where there is no memory pressure(memory pressure is defined as 80% of the maximum). Memory budgets can be set as a global limit or a per-channel limit(as of right now there is no way to specialize this limit for different channels). 
+
+All queues, and their associated schema and other metadata are packaged into channels, which has two sides: the highly templated and specialized side which is exposed to the callers, and the type-erased version which is exposed to the background thread. Therefore, all direct SQLite interactions are done opaquely through the type erased class by the background thread, which controls the raw SQLite pointer. 
+
+## C++ API Description
+Tools can call three different methods in the C++ code. 
+
+First, the primary data logging mechanism is logToDb. This method takes a template list of column names as the template, which are checked at compile time for being valid SQLite column names. The methods do not do any writing themselves, they take the data and enqueue it as quickly as possible into a buffer. The other arguments taken are the table name, and the tool id/message id pair, analagous to how the existing logger functions do it. 
+
+logToDbBulk is essentially the same as logToDb, except it takes iterators on the values, and streams them to the queues in bulk. 
+
+logToDbMetadata is a "slow path" text to text key-value logging mechanism, useful for logging one-time or infrequent messages(e.g. weights passed as arguments that stay the same across the run) or events. 
+
+## TCL API Description
+The TCL API is for the user to script how the logger will handle database logging. The setters also have corresponding getters. There are analagous C++ methods that these wrap around. 
+
+utl::start_log_db \[filename\]: Enables logging, runs logger db setup, and starts the background thread.
+
+utl::stop_log_db : stops background thread, cleans up and closes db logging.
+
+utl::set_db_log_global_max_mem \[bytes\]: Set overall max memory for pressure mechanism
+
+utl::set_db_log_per_channel_max_mem \[bytes\]: Set per-channel max memory for pressure mechanism
+
+utl::set_db_log_enabled \[tool name(e.g. GPL)\] \[true/false\]: Switch on/off per-tool logging. 
+
+## Performance Overhead
+Qualitatively acceptable. Measuring this in real-world use remains TODO. 

etc/DependencyInstaller.sh

@@ -1004,7 +1004,7 @@ _install_ubuntu_packages() {
         debhelper devscripts flex g++ gcc git groff lcov libbz2-dev libffi-dev libfl-dev \
         libgomp1 libomp-dev libpcre2-dev libreadline-dev pandoc \
         pkg-config python3-dev qt5-image-formats-plugins tcl tcl-dev \
-        tcllib unzip wget libyaml-cpp-dev zlib1g-dev tzdata
+        tcllib unzip wget libyaml-cpp-dev zlib1g-dev tzdata sqlite3 libsqlite3-dev
 
     local packages=()
     if _version_compare "$1" -ge "25.04"; then

etc/find_messages.py

@@ -60,6 +60,20 @@ def parse_args():
     re.VERBOSE | re.MULTILINE,
 )
 
+# Regex for logToDb / logToDbBulk calls (share the same id namespace).
+logtodb_regexp_c = re.compile(
+    r"""
+      (?:->|\.)                                        # deref
+      logToDb(?:Bulk)?                                  # logToDb or logToDbBulk
+      <.+?>                                            # template header
+      \s*\(\s*                                         # (
+      (?:utl::)?(?P<tool>[A-Z]{3})                     # tool
+      \s*,\s*                                          # ,
+      (?P<id>\d+)                                      # id
+    """,
+    re.VERBOSE | re.MULTILINE,
+)
+
 
 warn_regexp_tcl = re.compile(
     r"""
@@ -105,6 +119,24 @@ def scan_file(path, file_name, msgs):
 
         msgs[key].add(value)
 
+    if not file_name.endswith("tcl"):
+        for match in re.finditer(logtodb_regexp_c, lines):
+            tool = match.group("tool")
+            msg_id = int(match.group("id"))
+            key = "{} {:04d}".format(tool, msg_id)
+
+            line_num = lines[0 : match.start()].count("\n") + 1
+            position = "{}:{}".format(file_name, line_num)
+            file_link = os.path.join(path, file_name).strip("../").replace("\\", "/")
+            file_link = "https://github.com/The-OpenROAD-Project/OpenROAD/tree/master/{}#L{}".format(
+                file_link, line_num
+            )
+            value = "{:25} {:<50} DB_LOG {}".format(
+                position, "(database log)", file_link
+            )
+
+            msgs[key].add(value)
+
 
 def scan_dir(path, files, msgs):
     for file_name in files:

src/utl/BUILD

@@ -94,6 +94,7 @@ cc_library(
         "@boost.iterator",
         "@boost.random",
         "@spdlog",
+        "@sqlite3//:sqlite3",
         "@tcl_lang//:tcl",
     ],
 )

src/utl/CMakeLists.txt

@@ -58,6 +58,7 @@ target_include_directories(utl_lib
     include
   PRIVATE
     src
+    ${Boost_INCLUDE_DIRS}
     ${TCL_INCLUDE_PATH}
 )
 
@@ -68,8 +69,7 @@ target_link_libraries(utl_lib
     spdlog::spdlog
     ${TCL_LIBRARY}
     ${Boost_LIBRARIES}
-  PRIVATE
-    Boost::headers
+    sqlite3
 )
 
 target_sources(utl
@@ -83,10 +83,9 @@ target_include_directories(utl
     include
   PRIVATE
     src
+    ${Boost_INCLUDE_DIRS}
 )
 
-target_link_libraries(utl PRIVATE Boost::headers)
-
 target_compile_definitions(utl
   PUBLIC
     FMT_DEPRECATED_OSTREAM=1