← Problems86. Quarantine the records that will not parseMediumPySpark
00:00 / 25:00

Quarantine the records that will not parse

MediumNot attempted yetAsked at Deloitte, Uber, TCS

A partner posts one JSON object per payment. Some of them are not JSON at all, and the pipeline must not stop because of it.

Input

payloads record_id bigint body string -- a JSON object, in theory

A well-formed body looks like {"amount": 10.5, "currency": "INR"}.

Output

Every input row, with the columns:

  • record_id
  • status — 'VALID' or 'INVALID'
  • amount — the parsed amount on a valid row, null otherwise
  • currency — the parsed currency on a valid row, null otherwise

A row is VALID only when the body parses and both fields carry a value. Anything else is INVALID and carries neither field forward.

Example

record_idbodystatusamountcurrency
1{"amount": 10.5, "currency": "INR"}VALID10.5INR
2{"amount": 10.5}INVALIDnullnull
3not jsonINVALIDnullnull

Constraints

  • Declare the schema. Inferring it means reading the data first, and makes the output columns depend on which payloads happened to arrive.
  • Parsing is not validation. {"amount": 10.5} parses perfectly well.
  • 0.0 is an amount. A JSON null is not.
  • Nothing may throw. A batch of entirely unparseable bodies is a normal batch.
  • Row order is not graded. Column names and their order are.

Example 1one complete payload and one missing a field

Input · payloads
bodyrecord_id
{"amount": 10.5, "currency": "INR"}1
{"amount": 10.5}2
Output
record_idstatusamountcurrency
1VALID10.5INR
2INVALIDnullnull

Example 2text that is not JSON at all

Input · payloads
bodyrecord_id
not json3
{"amount": 2.0, "currency": "USD"}4
Output
record_idstatusamountcurrency
3INVALIDnullnull
4VALID2USD

Topics

json handlingdata quality

Similar problems

Community-reported interview topic. Not an official company question and no affiliation is implied.

solution.py
Loading editor…
Draft not saved yet · Spaces 4 · UTF-8⌘↵ run ⌘⇧↵ submit
Nothing run yet

Run against the public tests, or submit to score against all of them.