i get data as a json string that i like to flat then unmerge but i also need to keep the hierarchy. By that, i mean that if i have the following json :
{ "main" : [ { "a" : "1" , "sub" : [ { "x" : 10} ,{ "x" : 11} ] } , { "a" : "2" , "sub" : [ { "x" : 20} ,{ "x" : 21} ] } , { "a" : "3" } ]}
i like to retrieve the following table in pipeline pilot:
# a x
1 1 10
2 1 11
3 2 20
4 2 21
5 3
If i use the component 'data from JSON' then 'flatten hierarchy' , i get :
# a x
1 1 10
2 2 11
3 3 20
4 21
which is not what i want.
I make a script a python that do the right thing, but it consumes too much memory and sometimes pipeline pilot stops working with the following error 'INTERNAL ERROR: job information could not be obtained'
how could i achieve the same result (=flatten but keep the hierarchy) as in my python script (protocol in attachment) but in pipeline pilot ? i tried by 'Include Array Info in Path' then 'unpivot' , but then it is hard to merge again with the right parents.
thanks for your help!