Distinguishing between omitted and null values

See https://stackoverflow.com/questions/48510822/deserialize-json-distinguising-missing-and-null-values for context. I am surprised that out of the box the play-json read macros do not have an easy way to distinguish between {"foo": null} and {} – both of these cases would deserialize to None for an Option[String] property, whereas I see these two cases as having explicit intent especially in regards to an update (e.g. “I want to unset the value of foo”, vs. “I don’t want to change the value of foo”). As suggested in the StackOverflow thread above, using Option[Option[A]] is one way to represent this intent, but it would require a fair amount of custom code to implement. Best case scenario this could be configured in the JsonConfiguration so that we would only need to import this custom code to provide this functionality to our classes. However, I feel that the play-json team should have some opinion about this scenario, and they could likely represent it better in the core library. Has this been discussed internally? I am curious what the core members think about this.

Thanks for the question! We try to keep such discussions in the open, but indeed sometimes things get lost in chat so good to bring it up here.

This is a subtle issue: the question is how to represent a possibly-missing value on the Scala side. If we represent it as an Option[T], then this means we would be introducing either null or Some(null) values, which is not very idiomatic, and would definitely lead to NullPointerException problems.

Modeling this difference as an Option[Option[T]] (or wrapping it in your own multi-state class) as mentioned on stackoverflow would make sense from that perspective, but indeed is not so easy to implement.

Allowing customizing the macro to determine how to deserialize missing and null values would be an attractive way to make this easier, and has been discussed before but so far nobody has taken the initiative to implement this.

Note that cchantep, who also participated in the linked stackoverflow question, is also one of the most active play-json committers ;).

Thanks @raboof – I did not notice that issue was related to this before, so nice to have a link to that. My primary intent with the question was to ensure that this proposal wasn’t discussed and rejected for some reason before. Sounds like there are no red flags about this approach, just that it is not easy to accomplish at the moment especially in relation to macros. I will play around with this and see if I can get something working locally. I have no experience with Scala macros so I might be slow to provide help there, but I can take a look at the play-json macros implementation as well.

If you wanted to implement this yourself, you might be able to do something with Magnolia, which eliminates a lot of the tricky bits of implementing macros for case classes.

A couple years ago I wrote a proof of concept of implementing Reads and Writes macros with Magnolia: https://github.com/gmethvin/play-json-magnolia/blob/master/core/src/main/scala/io/methvin/play/json/MagnoliaJson.scala. The code is a bit ugly because I had to add some special cases specifically because Option is treated specially, but maybe this could serve as a starting point for future exploration.