Documentation

You are viewing the documentation for the 3.0.2 release. The latest stable release series is 3.0.x.

§JSON transformers

Please note this documentation was initially published as an article by Pascal Voitot (@mandubian) on mandubian.com

Now you should know how to validate JSON and convert into any structure you can write in Scala and back to JSON. But as soon as I’ve begun to use those combinators to write web applications, I almost immediately encountered a case : read JSON from network, validate it and convert it into… JSON.

§Introducing JSON coast-to-coast design

§Are we doomed to convert JSON to OO?

For a few years now, in almost all web frameworks (except recent JavaScript server side stuff maybe in which JSON is the default data structure), we have been used to get JSON from network and convert JSON (or even POST/GET data) into OO structures such as classes (or case classes in Scala). Why?

§Is OO conversion really the default use case?

In many cases, you don’t really need to perform any real business logic with data but validating/transforming before storing or after extracting. Let’s take the CRUD case:

So, generally, for CRUD ops, you convert JSON into a OO structure just because the frameworks are only able to speak OO.

I don’t say or pretend you shouldn’t use JSON to OO conversion but maybe this is not the most common case and we should keep conversion to OO only when we have real business logic to fulfill.

§New tech players change the way of manipulating JSON

Besides this fact, we have some new DB types such as MongoDB (or CouchDB) accepting document structured data looking almost like JSON trees (_isn’t BSON, Binary JSON?_).

With these DB types, we also have new great tools such as ReactiveMongo which provides reactive environment to stream data to and from Mongo in a very natural way.

I’ve been working with Stephane Godbillon to integrate ReactiveMongo with Play2.1 while writing the Play2-ReactiveMongo module. Besides Mongo facilities for Play2.1, this module provides Json To/From BSON conversion typeclasses.

So it means you can manipulate JSON flows to and from DB directly without even converting into OO.

§JSON coast-to-coast design

Taking this into account, we can easily imagine the following:

This is exactly the same case when serving data from DB:

In this context, we can easily imagine manipulating a flow of JSON data from client to DB and back without any (explicit) transformation in anything else than JSON.
Naturally, when you plug this transformation flow on reactive infrastructure provided by Play2.1, it suddenly opens new horizons.

This is the so-called (by me) JSON coast-to-coast design:

  • Don’t consider JSON data chunk by chunk but as a continuous flow of data from client to DB (or else) through server,
  • Treat the JSON flow like a pipe that you connect to others pipes while applying modifications, transformations alongside,
  • Treat the flow in a fully asynchronous/non-blocking way.

This is also one of the reason of being of Play2.1 reactive architecture…
I believe considering your app through the prism of flows of data changes drastically the way you design your web apps in general. It may also open new functional scopes that fit today’s webapps requirements quite better than classic architecture. Anyway, this is not the subject here ;)

So, as you have deduced by yourself, to be able to manipulate Json flows based on validation and transformation directly, we needed some new tools. JSON combinators were good candidates but they are a bit too generic.
That’s why we have created some specialized combinators and API called JSON transformers to do that.

§JSON transformers are Reads[T <: JsValue]

Keep in mind that a Reads[A <: JsValue] is able to transform and not only to read/validate

§Use JsValue.transform instead of JsValue.validate

We have provided a function helper in JsValue to help people consider a Reads[T] is a transformer and not only a validator:

JsValue.transform[A <: JsValue](reads: Reads[A]): JsResult[A]

This is exactly the same JsValue.validate(reads)

§The details

In the code samples below, we’ll use the following JSON:

{
  "key1" : "value1",
  "key2" : {
    "key21" : 123,
    "key22" : true,
    "key23" : [ "alpha", "beta", "gamma"],
    "key24" : {
      "key241" : 234.123,
      "key242" : "value242"
    }
  },
  "key3" : 234
}

§Case 1: Pick JSON value in JsPath

§Pick value as JsValue

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key23).json.pick

scala> json.transform(jsonTransformer)
res9: play.api.libs.json.JsResult[play.api.libs.json.JsValue] = 
  JsSuccess(
    ["alpha","beta","gamma"],
    /key2/key23
  )

§(__ \ 'key2 \ 'key23).json...

§(__ \ 'key2 \ 'key23).json.pick

§JsSuccess(["alpha","beta","gamma"],/key2/key23)

Reminder
jsPath.json.pick gets ONLY the value inside the JsPath

§Pick value as Type

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key23).json.pick[JsArray]

scala> json.transform(jsonTransformer)
res10: play.api.libs.json.JsResult[play.api.libs.json.JsArray] = 
  JsSuccess(
    ["alpha","beta","gamma"],
    /key2/key23
  )

§(__ \ 'key2 \ 'key23).json.pick[JsArray]

Reminder
jsPath.json.pick[T <: JsValue] extracts ONLY the typed value inside the JsPath

§Case 2: Pick branch following JsPath

§Pick branch as JsValue

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key24 \ 'key241).json.pickBranch

scala> json.transform(jsonTransformer)
res11: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
  {
    "key2": {
      "key24":{
        "key241":234.123
      }
    }
  },
  /key2/key24/key241
  )

§(__ \ 'key2 \ 'key23).json.pickBranch

§{"key2":{"key24":{"key242":"value242"}}}

Reminder:
jsPath.json.pickBranch extracts the single branch down to JsPath + the value inside JsPath

§Case 3: Copy a value from input JsPath into a new JsPath

import play.api.libs.json._

val jsonTransformer = (__ \ 'key25 \ 'key251).json.copyFrom( (__ \ 'key2 \ 'key21).json.pick )

scala> json.transform(jsonTransformer)
res12: play.api.libs.json.JsResult[play.api.libs.json.JsObject] 
  JsSuccess( 
    {
      "key25":{
        "key251":123
      }
    },
    /key2/key21
  )

§(__ \ 'key25 \ 'key251).json.copyFrom( reads: Reads[A <: JsValue] )

§{"key25":{"key251":123}}

Reminder:
jsPath.json.copyFrom(Reads[A <: JsValue]) reads value from input JSON and creates a new branch with result as leaf

§Case 4: Copy full input Json & update a branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key24).json.update( 
  __.read[JsObject].map{ o => o ++ Json.obj( "field243" -> "coucou" ) }
)

scala> json.transform(jsonTransformer)
res13: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key1":"value1",
      "key2":{
        "key21":123,
        "key22":true,
        "key23":["alpha","beta","gamma"],
        "key24":{
          "key241":234.123,
          "key242":"value242",
          "field243":"coucou"
        }
      },
      "key3":234
    },
  )

§(__ \ 'key2).json.update(reads: Reads[A < JsValue])

§(__ \ 'key2 \ 'key24).json.update(reads) does 3 things:

§JsSuccess({…},)

Reminder:
jsPath.json.update(Reads[A <: JsValue]) only works for JsObject, copies full input JsObject and updates jsPath with provided Reads[A <: JsValue]

§Case 5: Put a given value in a new branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key24 \ 'key241).json.put(JsNumber(456))

scala> json.transform(jsonTransformer)
res14: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key24":{
        "key241":456
      }
    },
  )

§(__ \ 'key24 \ 'key241).json.put( a: => JsValue )

§(__ \ 'key24 \ 'key241).json.put( a: => JsValue )

§jsPath.json.put( a: => JsValue )

§jsPath.json.put

**Reminder: **
jsPath.json.put( a: => Jsvalue ) creates a new branch with a given value without taking into account input JSON

§Case 6: Prune a branch from input JSON

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2 \ 'key22).json.prune

scala> json.transform(jsonTransformer)
res15: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key1":"value1",
      "key3":234,
      "key2":{
        "key21":123,
        "key23":["alpha","beta","gamma"],
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2/key22/key22
  )

§(__ \ 'key2 \ 'key22).json.prune

§(__ \ 'key2 \ 'key22).json.prune

Please note the resulting JsObject hasn’t same keys order as input JsObject. This is due to the implementation of JsObject and to the merge mechanism. But this is not important since we have overridden JsObject.equals method to take this into account.

Reminder:
jsPath.json.prune only works with JsObject and removes given JsPath form input JSON)

Please note that:
- prune doesn’t work for recursive JsPath for the time being
- if prune doesn’t find any branch to delete, it doesn’t generate any error and returns unchanged JSON.

§More complicated cases

§Case 7: Pick a branch and update its content in 2 places

import play.api.libs.json._
import play.api.libs.json.Reads._

val jsonTransformer = (__ \ 'key2).json.pickBranch(
  (__ \ 'key21).json.update( 
    of[JsNumber].map{ case JsNumber(nb) => JsNumber(nb + 10) }
  ) andThen 
  (__ \ 'key23).json.update( 
    of[JsArray].map{ case JsArray(arr) => JsArray(arr :+ JsString("delta")) }
  )
)

scala> json.transform(jsonTransformer)
res16: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key2":{
        "key21":133,
        "key22":true,
        "key23":["alpha","beta","gamma","delta"],
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2
  )

§(__ \ 'key2).json.pickBranch(reads: Reads[A <: JsValue])

§(__ \ 'key21).json.update(reads: Reads[A <: JsValue])

§of[JsNumber]

§of[JsNumber].map{ case JsNumber(nb) => JsNumber(nb + 10) }

§andThen

§of[JsArray].map{ case JsArray(arr) => JsArray(arr :+ JsString("delta")

Please note the result is just the __ \ 'key2 branch since we picked only this branch

§Case 8: Pick a branch and prune a sub-branch

import play.api.libs.json._

val jsonTransformer = (__ \ 'key2).json.pickBranch(
  (__ \ 'key23).json.prune
)

scala> json.transform(jsonTransformer)
res18: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "key2":{
        "key21":123,
        "key22":true,
        "key24":{
          "key241":234.123,
          "key242":"value242"
        }
      }
    },
    /key2/key23
  )

§(__ \ 'key2).json.pickBranch(reads: Reads[A <: JsValue])

§(__ \ 'key23).json.prune

Please remark the result is just the __ \ 'key2 branch without key23 field.

§What about combinators?

I stop there before it becomes boring (if not yet)…

Just keep in mind that you have now a huge toolkit to create generic JSON transformers. You can compose, map, flatmap transformers together into other transformers. So possibilities are almost infinite.

But there is a final point to treat: mixing those great new JSON transformers with previously presented Reads combinators. This is quite trivial as JSON transformers are just Reads[A <: JsValue]

Let’s demonstrate by writing a Gizmo to Gremlin JSON transformer.

Here is Gizmo:

val gizmo = Json.obj(
  "name" -> "gizmo",
  "description" -> Json.obj(
    "features" -> Json.arr( "hairy", "cute", "gentle"),
    "size" -> 10,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> Json.obj(
      "wet" -> "multiplies",
      "feed after midnight" -> "becomes gremlin"
    )
  ),
  "loves" -> "all"
)

Here is Gremlin:

val gremlin = Json.obj(
  "name" -> "gremlin",
  "description" -> Json.obj(
    "features" -> Json.arr("skinny", "ugly", "evil"),
    "size" -> 30,
    "sex" -> "undefined",
    "life_expectancy" -> "very old",
    "danger" -> "always"
  ),
  "hates" -> "all"
)

Ok let’s write a JSON transformer to do this transformation

import play.api.libs.json._
import play.api.libs.json.Reads._
import play.api.libs.functional.syntax._

val gizmo2gremlin = (
  (__ \ 'name).json.put(JsString("gremlin")) and
  (__ \ 'description).json.pickBranch(
    (__ \ 'size).json.update( of[JsNumber].map{ case JsNumber(size) => JsNumber(size * 3) } ) and
    (__ \ 'features).json.put( Json.arr("skinny", "ugly", "evil") ) and
    (__ \ 'danger).json.put(JsString("always"))
    reduce
  ) and
  (__ \ 'hates).json.copyFrom( (__ \ 'loves).json.pick )
) reduce

scala> gizmo.transform(gizmo2gremlin)
res22: play.api.libs.json.JsResult[play.api.libs.json.JsObject] = 
  JsSuccess(
    {
      "name":"gremlin",
      "description":{
        "features":["skinny","ugly","evil"],
        "size":30,
        "sex":"undefined",
        "life_expectancy":
        "very old","danger":"always"
      },
      "hates":"all"
    },
  )

Here we are ;)
I’m not going to explain all of this because you should be able to understand now.
Just remark:

§(__ \ 'features).json.put(…) is after (__ \ 'size).json.update so that it overwrites original (__ \ 'features)

§(Reads[JsObject] and Reads[JsObject]) reduce

Next: Working with XML