Problem with json decoding

Shyam

Hello again. I am writing a small program to get weather forecast information to do a risk evaluation if certain condition are met.

The problem is that the json decoding doesn't work as intendend(or maybe i am making something wrong because it output different numbers than the one written in the json string) so before writing my own parser to get the data from the json string i wanted to ask what could the problem be with the following code:


local net = require "net"
local json= require "json" 


function readValueFromTable(t)
  for k,v in pairs(t) do 
    if type(v)~="table" then print(k.."----"..v) else
       print("--------------------\n"..k.." is a table within a table\n--------------------")
      readValueFromTable(v)
    end
  end
end  

--get request of json from free open-meteo api, also create a local txt file containing the json
local client = net.Http("https://marine-api.open-meteo.com")
local task=client:get("/v1/marine?latitude=42.80775&longitude=10.73688&hourly=wave_height,wave_direction,wave_period&wind_speed_unit=kn&forecast_days=1")
local json_decoded=nil
task.after=function(self, response)
  local jsonObj= response.content
  local txtJson=sys.File(sys.currentdir.."\\test.txt")
  txtJson:open("write")
  txtJson:write(jsonObj)
  txtJson:close()
  json_decoded=json.decode(jsonObj)
end

waitall()

--print all the contet of the received json
for k,v in pairs(json_decoded) do
  if type(v)~="table" then print(k.."----"..v) 
  else 
  print("--------------------\n"..k.." is a table within a jsonObject\n--------------------")
  readValueFromTable(v)
  end
end

Shyam

It's clear that some kind of conversion problem is happening with all the float number values and also with stuff like the wave_direction table that should be just one integer value for each index but instead it print two numbers separated by a comma.

Shyam

To add to this problem, by adding &format=xlsx to the GET string it is possible to obtain the file data in that format.

In the browser there is button that do that and is interpreted by the broswer as a download and get the file. Using the developer tool of the browser i can see it is a GET request and so i wrote this piece of code but while it definetly write something, because the file have some data in it, it doesn't allow me to open it saying that the format is not valid.

local net = require "net"


--by adding the "&format=xlsx" it is possible to receive the data in a different format

local client = net.Http("https://marine-api.open-meteo.com")

local task=client:get("/v1/marine?latitude=54.544587&longitude=10.227487&hourly=wave_height,wave_direction,wave_period,wind_wave_height,wind_wave_direction,wind_wave_period&wind_speed_unit=kn&forecast_days=1&format=xlsx")
task.after=function(self, response)
local file=sys.File("C:\\Program Files\\luaRT\\LuaRT-Studio\\Projects\\Some Gui App\\Meteo\\test.xlsx")
file:open("write")
file:write(response.content)
file:close()
end

waitall()

If instead i format it in csv and create a .csv file i can just create the file on my pc as i was downloading it, the problem is only with the xlsx extension.

if possible i would like to understand how to fix both codes cause in the future i want to do more stuff with http and socket library so the more i understand about this the better

Thanks a lot for reading all of this, hope the solution is easy to find.

Samir

You are right. I have found a bug with float numbers parsing when using json.encode() function.
It is now fixed (and will be available in next LuaRT release)

I will check the xlsx issue now but not sure it's LuaRT related

Samir

For the xlsx part.

You souldn't use Http:get() method to download a file : the end of file cannot be guaranteed in this scenario (that's why Excel cannot open the file as it is not complete)

You have to use the Http:download() function to send a GET request and download its result.

But even with that, the URI provided cannot result in a valid filename, so the Http:download() method won't be able to save data to disk.

I have updated the function to use a default filename in that case, and the possibility to choose the filename of the downloaded file. It will be available in next LuaRT release.

Many thanks for your help in improving the net module !

Shyam

Very glad to know i could be helpful somehow.

I forget to add regarding the xlsm that the :download() method was not working that's why i resorted to just use get().
Probably should have told that before for a more clear bug report.

One last thing, regarding the JSON, was the bug only about the floating numbers?

Because, as i said, and probably you noticed if you run the code, also integer number where not correctly decoded. Wanted to be sure that it's part of the same bug, cause otherwise you would need to also investigate that.

Thank you so much again for your job, and your patience.

Hope to see the new update soon.

Samir

Yes it's corrected for both integers and floating numbers 👍️

Shyam

Samir 👌