1. Home
2. Working with Play
3. Play for Scala developers
4. Main concepts
5. Working with Json

§JSON transformers

Please note this documentation was initially published as an article by Pascal Voitot (@mandubian) on mandubian.com

Now you should know how to validate JSON and convert into any structure you can write in Scala and back to JSON. But as soon as I’ve begun to use those combinators to write web applications, I almost immediately encountered a case : read JSON from network, validate it and convert it into… JSON.

§Introducing JSON coast-to-coast design

§Are we doomed to convert JSON to OO?

For a few years now, in almost all web frameworks (except recent JavaScript server side stuff maybe in which JSON is the default data structure), we have been used to get JSON from network and convert JSON (or even POST/GET data) into OO structures such as classes (or case classes in Scala). Why?

• For a good reason : OO structures are “language-native” and allows manipulating data with respect to your business logic in a seamless way while ensuring isolation of business logic from web layers.
• For a more questionable reason : ORM frameworks talk to DB only with OO structures and we have (kind of) convinced ourselves that it was impossible to do else… with the well-known good & bad features of ORMs… (not here to criticize those stuff)

§Is OO conversion really the default use case?

In many cases, you don’t really need to perform any real business logic with data but validating/transforming before storing or after extracting. Let’s take the CRUD case:

• You just get the data from the network, validate them a bit and insert/update into DB.
• In the other way, you just retrieve data from DB and send them outside.

So, generally, for CRUD ops, you convert JSON into a OO structure just because the frameworks are only able to speak OO.

I don’t say or pretend you shouldn’t use JSON to OO conversion but maybe this is not the most common case and we should keep conversion to OO only when we have real business logic to fulfill.

§New tech players change the way of manipulating JSON

Besides this fact, we have some new DB types such as MongoDB (or CouchDB) accepting document structured data looking almost like JSON trees (_isn’t BSON, Binary JSON?_).

With these DB types, we also have new great tools such as ReactiveMongo which provides reactive environment to stream data to and from Mongo in a very natural way.

I’ve been working with Stephane Godbillon to integrate ReactiveMongo with Play2.1 while writing the Play2-ReactiveMongo module. Besides Mongo facilities for Play2.1, this module provides Json To/From BSON conversion typeclasses.

So it means you can manipulate JSON flows to and from DB directly without even converting into OO.

§JSON coast-to-coast design

Taking this into account, we can easily imagine the following:

• Receive JSON.
• Validate JSON.
• Transform JSON to fit expected DB document structure.
• Directly send JSON to DB (or somewhere else).

This is exactly the same case when serving data from DB:

• Extract some data from DB as JSON directly.
• Filter/transform this JSON to send only mandatory data in the format expected by the client (e.g you don’t want some secure info to go out).
• Directly send JSON to the client.

In this context, we can easily imagine manipulating a flow of JSON data from client to DB and back without any (explicit) transformation in anything else than JSON.
Naturally, when you plug this transformation flow on reactive infrastructure provided by Play2.1, it suddenly opens new horizons.

This is the so-called (by me) JSON coast-to-coast design:

• Don’t consider JSON data chunk by chunk but as a continuous flow of data from client to DB (or else) through server,
• Treat the JSON flow like a pipe that you connect to others pipes while applying modifications, transformations alongside,
• Treat the flow in a fully asynchronous/non-blocking way.

This is also one of the reason of being of Play2.1 reactive architecture…
I believe considering your app through the prism of flows of data changes drastically the way you design your web apps in general. It may also open new functional scopes that fit today’s webapps requirements quite better than classic architecture. Anyway, this is not the subject here ;)

So, as you have deduced by yourself, to be able to manipulate Json flows based on validation and transformation directly, we needed some new tools. JSON combinators were good candidates but they are a bit too generic.
That’s why we have created some specialized combinators and API called JSON transformers to do that.

§JSON transformers are Reads[T <: JsValue]

• You may tell JSON transformers are just f:JSON => JSON.
• So a JSON transformer could be simply a Writes[A <: JsValue].
• But, a JSON transformer is not only a function: as we said, we also want to validate JSON while transforming it.
• As a consequence, a JSON transformer is a Reads[A <: JsValue].

Keep in mind that a Reads[A <: JsValue] is able to transform and not only to read/validate

§Use JsValue.transform instead of JsValue.validate

We have provided a function helper in JsValue to help people consider a Reads[T] is a transformer and not only a validator:

JsValue.transform[A <: JsValue](reads: Reads[A]): JsResult[A]

This is exactly the same JsValue.validate(reads)

§The details

In the code samples below, we’ll use the following JSON:

{
  "key1" : "value1",
  "key2" : {
    "key21" : 123,
    "key22" : true,
    "key23" : [ "alpha", "beta", "gamma"],
    "key24" : {
      "key241" : 234.123,
      "key242" : "value242"
    }
  },
  "key3" : 234
}

§Case 1: Pick JSON value in JsPath

§Pick value as JsValue

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key23).json.pick

scala> json.transform(jsonTransformer)
res9: play.api.libs.json.JsResult[play.api.libs.json.JsValue] = 
  JsSuccess(
    ["alpha","beta","gamma"],
    /key2/key23
  )

§(__ \ 'key2 \ 'key23).json...

• All JSON transformers are in JsPath.json.

§(__ \ 'key2 \ 'key23).json.pick

• pick is a Reads[JsValue] which picks the value IN the given JsPath. Here ["alpha","beta","gamma"]

§JsSuccess(["alpha","beta","gamma"],/key2/key23)

• This is a simply successful JsResult.
• For info, /key2/key23 represents the JsPath where data were read but don’t care about it, it’s mainly used by Play API to compose JsResult(s)).
• ["alpha","beta","gamma"] is just due to the fact that we have overridden toString.

Reminder
jsPath.json.pick gets ONLY the value inside the JsPath

§Pick value as Type

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key23).json.pick[JsArray]

scala> json.transform(jsonTransformer)
res10: play.api.libs.json.JsResult[play.api.libs.json.JsArray] = 
  JsSuccess(
    ["alpha","beta","gamma"],
    /key2/key23
  )

§(__ \ 'key2 \ 'key23).json.pick[JsArray]

• pick[T] is a Reads[T <: JsValue] which picks the value (as a JsArray in our case) IN the given JsPath

Reminder
jsPath.json.pick[T <: JsValue] extracts ONLY the typed value inside the JsPath

§Case 2: Pick branch following JsPath

§Pick branch as JsValue

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key24 \ 'key241).json.pickBranch

scala> json.transform(jsonTransformer)
res11: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
  {
    "key2": {
      "key24":{
        "key241":234.123
      }
    }
  },
  /key2/key24/key241
  )

§(__ \ 'key2 \ 'key23).json.pickBranch

• pickBranch is a Reads[JsValue] which picks the branch from root to given JsPath

§{"key2":{"key24":{"key242":"value242"}}}

• The result is the branch from root to given JsPath including the JsValue in JsPath

Reminder:
jsPath.json.pickBranch extracts the single branch down to JsPath + the value inside JsPath

§Case 3: Copy a value from input JsPath into a new JsPath

import play.api.libs.json._

val jsonTransformer = (__ \ 'key25 \ 'key251).json.copyFrom( (__ \ 'key2 \ 'key21).json.pick )

scala> json.transform(jsonTransformer)
res12: play.api.libs.json.JsResult[play.api.libs.json.JsObject] 
  JsSuccess( 
    {
      "key25":{
        "key251":123
      }
    },
    /key2/key21
  )

§(__ \ 'key25 \ 'key251).json.copyFrom( reads: Reads[A <: JsValue] )

• copyFrom is a Reads[JsValue]
• copyFrom reads the JsValue from input JSON using provided Reads[A]
• copyFrom copies this extracted JsValue as the leaf of a new branch corresponding to given JsPath

§{"key25":{"key251":123}}

• copyFrom reads value 123
• copyFrom copies this value into new branch (__ \ 'key25 \ 'key251)

Reminder:
jsPath.json.copyFrom(Reads[A <: JsValue]) reads value from input JSON and creates a new branch with result as leaf

§Case 4: Copy full input Json & update a branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key24).json.update( 
  __.read[JsObject].map{ o => o ++ Json.obj( "field243" -> "coucou" ) }
)

scala> json.transform(jsonTransformer)
res13: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key1":"value1",
      "key2":{
        "key21":123,
        "key22":true,
        "key23":["alpha","beta","gamma"],
        "key24":{
          "key241":234.123,
          "key242":"value242",
          "field243":"coucou"
        }
      },
      "key3":234
    },
  )

§(__ \ 'key2).json.update(reads: Reads[A < JsValue])

• Is a Reads[JsObject]

§(__ \ 'key2 \ 'key24).json.update(reads) does 3 things:

• Extracts value from input JSON at JsPath (__ \ 'key2 \ 'key24).
• Applies reads on this relative value and re-creates a branch (__ \ 'key2 \ 'key24) adding result of reads as leaf.
• Merges this branch with full input JSON replacing existing branch (so it works only with input JsObject and not other type of JsValue).

§JsSuccess({…},)

• Just for info, there is no JsPath as 2nd parameter there because the JSON manipulation was done from Root JsPath

Reminder:
jsPath.json.update(Reads[A <: JsValue]) only works for JsObject, copies full input JsObject and updates jsPath with provided Reads[A <: JsValue]

§Case 5: Put a given value in a new branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key24 \ 'key241).json.put(JsNumber(456))

scala> json.transform(jsonTransformer)
res14: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key24":{
        "key241":456
      }
    },
  )

§(__ \ 'key24 \ 'key241).json.put( a: => JsValue )

• Is a Reads[JsObject]

§(__ \ 'key24 \ 'key241).json.put( a: => JsValue )

• Creates a new branch (__ \ 'key24 \ 'key241)
• Puts a as leaf of this branch.

§jsPath.json.put( a: => JsValue )

• Takes a JsValue argument passed by name allowing to pass even a closure to it.

§jsPath.json.put

• Does not care at all about input JSON.
• Simply replace input JSON by given value.

**Reminder: **
jsPath.json.put( a: => Jsvalue ) creates a new branch with a given value without taking into account input JSON

§Case 6: Prune a branch from input JSON

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key22).json.prune

scala> json.transform(jsonTransformer)
res15: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key1":"value1",
      "key3":234,
      "key2":{
        "key21":123,
        "key23":["alpha","beta","gamma"],
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2/key22/key22
  )

§(__ \ 'key2 \ 'key22).json.prune

• Is a Reads[JsObject] that works only with JsObject

§(__ \ 'key2 \ 'key22).json.prune

• Removes given JsPath from input JSON (key22 has disappeared under key2)

Please note the resulting JsObject hasn’t same keys order as input JsObject. This is due to the implementation of JsObject and to the merge mechanism. But this is not important since we have overridden JsObject.equals method to take this into account.

Reminder:
jsPath.json.prune only works with JsObject and removes given JsPath form input JSON)

Please note that:
- prune doesn’t work for recursive JsPath for the time being
- if prune doesn’t find any branch to delete, it doesn’t generate any error and returns unchanged JSON.

§More complicated cases

§Case 7: Pick a branch and update its content in 2 places

import play.api.libs.json._
import play.api.libs.json.Reads._

val jsonTransformer = (__ \ 'key2).json.pickBranch(
  (__ \ 'key21).json.update( 
    of[JsNumber].map{ case JsNumber(nb) => JsNumber(nb + 10) }
  ) andThen 
  (__ \ 'key23).json.update( 
    of[JsArray].map{ case JsArray(arr) => JsArray(arr :+ JsString("delta")) }
  )
)

scala> json.transform(jsonTransformer)
res16: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key2":{
        "key21":133,
        "key22":true,
        "key23":["alpha","beta","gamma","delta"],
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2
  )

§(__ \ 'key2).json.pickBranch(reads: Reads[A <: JsValue])

• Extracts branch __ \ 'key2 from input JSON and applies reads to the relative leaf of this branch (only to the content).

§(__ \ 'key21).json.update(reads: Reads[A <: JsValue])

• Updates (__ \ 'key21) branch.

§of[JsNumber]

• Is just a Reads[JsNumber].
• Extracts a JsNumber from (__ \ 'key21).

§of[JsNumber].map{ case JsNumber(nb) => JsNumber(nb + 10) }

• Reads a JsNumber (_value 123_ in __ \ 'key21).
• Uses Reads[A].map to increase it by 10 (in immutable way naturally).

§andThen

• Is just the composition of 2 Reads[A].
• First reads is applied and then result is piped to second reads.

§of[JsArray].map{ case JsArray(arr) => JsArray(arr :+ JsString("delta")

• Reads a JsArray (_value [alpha, beta, gamma] in __ \ 'key23_).
• Uses Reads[A].map to append JsString("delta") to it.

Please note the result is just the __ \ 'key2 branch since we picked only this branch

§Case 8: Pick a branch and prune a sub-branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2).json.pickBranch(
  (__ \ 'key23).json.prune
)

scala> json.transform(jsonTransformer)
res18: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key2":{
        "key21":123,
        "key22":true,
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2/key23
  )

§(__ \ 'key2).json.pickBranch(reads: Reads[A <: JsValue])

• Extracts branch __ \ 'key2 from input JSON and applies reads to the relative leaf of this branch (only to the content).

§(__ \ 'key23).json.prune

• Removes branch __ \ 'key23 from relative JSON

Please remark the result is just the __ \ 'key2 branch without key23 field.

§What about combinators?

I stop there before it becomes boring (if not yet)…

Just keep in mind that you have now a huge toolkit to create generic JSON transformers. You can compose, map, flatmap transformers together into other transformers. So possibilities are almost infinite.

But there is a final point to treat: mixing those great new JSON transformers with previously presented Reads combinators. This is quite trivial as JSON transformers are just Reads[A <: JsValue]

Let’s demonstrate by writing a Gizmo to Gremlin JSON transformer.

Here is Gizmo:

val gizmo = Json.obj(
  "name" -> "gizmo",
  "description" -> Json.obj(
    "features" -> Json.arr( "hairy", "cute", "gentle"),
    "size" -> 10,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> Json.obj(
      "wet" -> "multiplies",
      "feed after midnight" -> "becomes gremlin"
    )
  ),
  "loves" -> "all"
)

Here is Gremlin:

val gremlin = Json.obj(
  "name" -> "gremlin",
  "description" -> Json.obj(
    "features" -> Json.arr("skinny", "ugly", "evil"),
    "size" -> 30,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> "always"
  ),
  "hates" -> "all"
)

Ok let’s write a JSON transformer to do this transformation

import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._

val gizmo2gremlin = (
  (__ \ 'name).json.put(JsString("gremlin")) and
  (__ \ 'description).json.pickBranch(
    (__ \ 'size).json.update( of[JsNumber].map{ case JsNumber(size) => JsNumber(size * 3) } ) and
    (__ \ 'features).json.put( Json.arr("skinny", "ugly", "evil") ) and
    (__ \ 'danger).json.put(JsString("always"))
    reduce
  ) and
  (__ \ 'hates).json.copyFrom( (__ \ 'loves).json.pick )
) reduce

scala> gizmo.transform(gizmo2gremlin)
res22: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "name":"gremlin",
      "description":{
        "features":["skinny","ugly","evil"],
        "size":30,
        "sex":"undefined",
        "life_expectancy":
        "very old","danger":"always"
      },
      "hates":"all"
    },
  )

Here we are ;)
I’m not going to explain all of this because you should be able to understand now.
Just remark:

§(__ \ 'features).json.put(…) is after (__ \ 'size).json.update so that it overwrites original (__ \ 'features)

§(Reads[JsObject] and Reads[JsObject]) reduce

• It merges results of both Reads[JsObject] (JsObject ++ JsObject)
• It also applies the same JSON to both Reads[JsObject] unlike andThen which injects the result of the first reads into second one.

Next: JSON Macro Inception