Think about splitting hpgsql into a core package with a lower level API and other packages

[mzabani]

There have been three mentions of how it'd be better if hpgsql wasn't so batteries-included, and provided a smaller core for other software to build on.

• On reddit here and here
• For PostgREST

While it's not entirely clear to me what this separation would entail, some things are:

• This core would take SQL statements as either ByteString or Text, and the Query type would not be in it, including all the parsing and template haskell/quasiquoters (would also not be in it).
  • This means this core would take dollar-numbered query arguments like postgres expects them, and would not take more than one statement in the same query.

The most basic core abstraction we have, which would very likely be in this core library, is:

-- | After sending one or more queries to the backend, run this function for each query to fetch that query's results.
-- You must call the returned IO function and consume the returned Stream completely until you get to the
-- `Either ErrorResponse CommandComplete` object.
-- For non-row returning statements like INSERT, DELETE, and UPDATE, the returned Stream will be empty but the execution status
-- will still be available in the Stream's result.
-- This function can also "consume results" of a `COPY FROM STDIN` statement, which essentially means it can receive the
-- control messages of that starting from any possible state.
-- By following these rules you will always keep the internal connection's state healthy, even in the presence of concurrency
-- and asynchronous exceptions.
consumeResults ::
  HPgConnection ->
  QueryId ->
  IO (Maybe (Either3 NoData RowDescription CopyInResponse), Stream (Of DataRow) IO (Either ErrorResponse CommandComplete))
consumeResults conn qryId = do

This still demands internally-assigned QueryIds (which are important for tracking things related to thread safety) which maybe users would prefer not to have? This raises another question: how much of thread-safety and interruption-safety should be in the core library? These things might not be easy to add on top of an API that is too basic.

Also, decoding DataRow into fields/rows is a lot of work for consumers, so maybe field and row decoders/encoders should also be part of this core package, just maybe not the typeclasses (FromPgField, ToPgField, FromPgRow, ToPgRow) currently associated with them.

I think it's best to hear feedback from potential users for this.

[echatav]

The interface for postgresql-libpq is very similar to that of libpq itself. The most important functions are exec and execParams which both return Results. Then with a Result in hand you can use getvalue to return a particular cell in the result based on its row & column number. A drop-in replacement would match these function signatures on the nose. One thing to note is that libpq lets you format parameters and demand return values be formatted as either binary or text Format. Squeal only uses binary format but I think that some other libraries might use text formatting.

[mzabani]

What if hpgsql-simple-compat, which already contains one module replicating a (tiny) part of postgresql-libpq's API, were to have cabal flags to hide the parts that replicate postgresql-simple-compat's and postgresql-query's APIs instead of being a separate package? And then of course evolve the same (or very similar) API as postgresql-libpq. Would that work for you?

And do you see value - maybe not now, but in the longer term - in having a different low level API available? The consumeResults function I exemplified in this github issue, for example, will return a Stream of data rows, which can allow you to do things with bounded memory, for example. Other differences might be not requiring so many methods to live in IO, for example.