Get raw ByteString out of a non-raw API client

[lrworth]

We have an API type representing a 3rd-party API that we want to use with servant-client. Our response type definitions only implement a subset of the response from the API.

Now, we want to use the same servant API definition to fetch the (unparsed) raw data from the 3rd-party service to store in a database for archiving. Using the existing stuff in servant-client-core, is it possible to access the raw response, including headers, without having to replicate the HasClient machinery to change how Verbs are handled? We'd like to be able to write the standard genericClient // endpoint /: param and get the full response back (or at least the raw body).

We also considered building an alternative runRequestAcceptStatus to log the raw responses in addition to the standard behaviour, but this would cause client requests to fail if the response could not be parsed into the declared type, even though we don't need the parsed version.

[JackKelly-Bellroy]

It seems like we want the ability to apply a type constructor or type family to the "results" of an API, something like:

-- Open family so that other packages can add combinators
type family AlterResult (f :: Type -> Type) api
type instance AlterResult f (x :> y) = x :> AlterResult f y
type instance AlterResult f (x :<|> y) = AlterResult f x :<|> AlterResult f y
type instance AlterResult f (Verb m s cs a) = Verb m s cs (f a)
-- etc

For our purposes, capturing the whole raw response would be pretty cool, but for ETL purposes even just getting the unparsed JSON might be enough. You could then invent some type families like:

type family Json a where
  Json a = Aeson.Value
type family Parsed a where
  Parsed a = a

We'd then like to be able to plumb these type families into generic-style records. The most promising way I found to do this was to define new "mode" types and GenericMode instances:

data AsApiF (f :: Type -> Type)
instance GenericMode (AsApiF f) where
  type AsApiF f :- api = AlterResult f api

data AsClientFT (f :: Type -> Type) (m :: Type -> Type)
instance GenericMode (AsClientFT f m) where
  type AsClientT f m :- api = AsClientT m :- AlterResult f api

It then seems like you'd be able to write variants of servant-client:Servant.Client.Generic.{genericClient,genericClientHoist} that plumb through an f. We attempted to write these but ran out of time, and went with other workarounds, which I'll detail below.

If plumbing the f through every endpoint is too difficult, being able to pluck the type of a record field by field name (in the AsApi mode) would also get us pretty close to what we want. We could then AlterResult on that, and then convert the old-style servant type to a client function.

---

Other workarounds:

The one we went with

data WithOriginalJson a = WithOriginalJson {value :: a, json :: Aeson.Value}
  deriving (Eq, Show, Generic)

instance (Aeson.FromJSON a) => Aeson.FromJSON (WithOriginalJson a) where
  parseJSON json = do
    value <- parseJSON json
    pure WithOriginalJson {..}

This pollutes the API definition but otherwise works well: you know that the responses parsed successfully and error handling is done by servant-client-core machinery. It feels a bit wasteful to carry around both the raw and parsed values, and might make debugging harder if we do start getting parse errors in response to a surprise API change.

Defer parsing

newtype Json a = Json Aeson.Value
  deriving newtype (Aeson.FromJSON)

unwrapRaw :: Json a -> Aeson.Value
unwrapRaw (Json j) = j

unwrapParsed :: (Aeson.FromJSON a) => Json a -> Either String a
unwrapParsed (Json j) = Aeson.parseEither Aeson.parseJSON j

This also pollutes the API definition but otherwise works pretty well. The main new advantage is that you don't have to carry around both structures. The upside/downside is that servant-client won't catch parse errors for you any more when you want to parse the response payload.

Parameterise the API record by the parsing mode

type data ParseMode = Raw | Parsed

-- Singleton for ParseMode
data SParseMode (mode :: ParseMode) where
  SRaw :: SParseMode Raw
  SParsed :: SParseMode Parsed

-- Implicit singleton for ParseMode
class SParseModeI mode where
  sparseModeI :: SParseMode

data Result (mode :: ParseMode) (a :: Type) where
  R :: Aeson.Value -> Result Raw a
  P :: a -> Result Parsed a

-- Can't write separate instances for `Result Raw a` and `Result Parsed a`;
-- GHC won't know that they're exhaustive.
instance (SParseModeI mode, FromJSON a) => FromJSON (Result mode a) where
  parseJSON = case sparseModeI @mode of
    SRaw -> pure . R
    SParsed -> fmap P . parseJSON

data Routes parseMode mode = Routes
  { endpoint1 :: mode :- "some" :> "api" :> Get '[JSON] (Result parseMode Endpoint1Response)
  , endpoint2 :: -- etc
  }

This very quickly needs singletons or a hand-rolled equivalent, still pollutes the API definition, and causes ambiguous type errors if you don't annotate every endpoint result.