Tesla Powerwall Local Connection broken with New Release

[jamesng]

Hi


For a few months now I've been connecting our SHAC / LogicMachine to our TeslaPowerwall to load-shift our electricity usage between the battery and the grid in response to the 30 minute electricity pricing provided by Amber Electric.  This has worked really well and we have seen our energy bills fall by about 50%.

Tesla pushed a firmware update the the Powerwall today (v21.20.2) which is causing an error with the local login script.  The remote login script for control of the Powerwall still works well).

Here is the local login script which is no longer working...

Code:

https = require('ssl.https')
json = require('json')
https.TIMEOUT = 5

loginURL = 'https://POWERWALLGATEWAY/api/login/Basic'
  loginBody = "{\"username\":\"customer\",\"password\":\"MYPASS\", \"email\":\"MY@EMAIL.COM\",\"force_sm_off\":false}"
  loginPostResponse = {}

  res, err, loginResponseHeaders, status = https.request
    {
      url = loginURL,
      method = 'POST',
      protocol = 'tlsv12',
      headers =
      {
        ['Accept'] = '*/*',
        ['Content-Type'] = 'application/json',
        ['Content-Length'] = loginBody:len()
      },
      source = ltn12.source.string(loginBody),     
      sink = ltn12.sink.table(loginPostResponse)
    }

   log(res, err, loginResponseHeaders) -- DEBUG

  if loginPostResponse then
   
    mmm_cookies = loginResponseHeaders['set-cookie']
    if mmm_cookies then
      token = mmm_cookies:match('(AuthCookie=[^;]+)'..';')
     
      log ("Successfully retrieved local Tesla Powerwall Login Token " .. token)
     
    end
    
  end 

Now, I get the following error

Code:

attempt to index global 'loginResponseHeaders' (a nil value)

and the debug line returns 

Code:

* arg: 1
  * nil
* arg: 2
  * string: closed
* arg: 3
  * nil

when trying to connect to the Powerwall with the above code.

Previously POWERWALLGATEWAY was a DNS name, however the PowerWall now drops the connection unless an IP address is used in a web browser.  I can connect ok to https://POWERWALLGATEWAY/api/login/Basic/ with a web browser to retrieve the token, however Logic Machine script no longer works.

Pointing a web browser at the Powerwall yields a NET::ERR_CERT_AUTHORITY_INVALID error, which one can click through, however the Logic Machine script can't get past this

The SSL certificate used by Tesla and discussed here https://github.com/vloschiavo/powerwall2. - but don't know how to work around this in the script.

Does anyone have another way that I can connect with logic machine using the IP address to get around this (SSL certificate?) issue?

Many thanks in advance

Kind Regards
James

[jamesng]

Further to my earlier post, the LUA code above was modelled on the CURL call below (which still works to retrieve AuthCookie / Login Token)

Code:

curl -s -k -i -c cookie.txt -X POST -H "Content-Type: application/json" -d "{\"username\":\"customer\",\"password\":\"MYPASS\", \"email\":\"MY@EMAIL.COM\",\"force_sm_off\":false}" "https://POWERWALLGATEWAYIP/api/login/Basic"

… which returns

HTTP/2 200
cache-control: no-cache, no-store
set-cookie: AuthCookie=Tm21w<SNIP>LQfX_Jfcvd8A==; Path=/
set-cookie: UserRecord=eyJlb<SNIP>ifQ==; Path=/
content-type: text/plain; charset=utf-8
content-length: 259
date: Fri, 02 Jul 2021 15:04:27 GMT

Can anyone see how the LUA version of this is different and why it would return a nil value


Kind Regards
James

[admin]

Try adding verbose flag to curl (-v) and post what you get. I suspect it might be possible that either HTTP/2 or TLS v1.3 is required.

[jamesng]

I got a very similar response with -v

curl -s -k -i -c -v cookie.txt -X POST -H "Content-Type: application/json" -d "{\"username\":\"customer\",\"password\":\"MYPASS\", \"email\":\"MY@EMAIL.COM\",\"force_sm_off\":false}" "https://POWERWALLGATEWAYIP/api/login/Basic"

… which returned

Code:

HTTP/2 200
cache-control: no-cache, no-store
set-cookie: AuthCookie=ZKHyR<SNIP>kJg==; Path=/
set-cookie: UserRecord=eyJl<SNIP>jAwIn0=; Path=/
content-type: text/plain; charset=utf-8
content-length: 260
date: Fri, 02 Jul 2021 16:01:48 GMT

{"email":"MY@EMAIL.COM,"firstname":"Tesla","lastname":"Energy","roles":["Home_Owner"],"token":"ZKHyR<SNIP>kJg==","provider":"Basic","loginTime":"2021-07-03T02:01:48.728093492+10:00"}%

Does the HTTP/2 mean a different protocol is needed to send the request with LogicMachine?

Kind Regards
James

[admin]

-v should be placed before -c, also try adding --http1.1 right after -v

[jamesng]

That returned a little more info

curl -s -k -i -v --http1.1 -c / ....

Code:

james@mac ~ % curl -s -k -i -v --http1.1 -c /Users/james/Desktop/cookie.txt -X POST -H "Content-Type: application/json" -d "{\"username\":\"customer\",\"password\":\"MYPASS\", \"email\":\"MY@EMAIL.COM\",\"force_sm_off\":false}" "https://POWERWALLGATEWAYIP/api/login/Basic"
*   Trying POWERWALLGATEWAYIP...
* TCP_NODELAY set
* Connected to POWERWALLGATEWAYIP (POWERWALLGATEWAYIP) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: C=US; ST=California; L=Palo Alto; O=Tesla; OU=Tesla Energy Products; CN=2c089aa83d18b2a35fb271e0a61d035f
*  start date: Jul 14 22:52:25 2018 GMT
*  expire date: Jul  8 22:52:25 2043 GMT
*  issuer: C=US; ST=California; L=Palo Alto; O=Tesla; OU=Tesla Energy Products; CN=2c089aa83d18b2a35fb271e0a61d035f
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> POST /api/login/Basic HTTP/1.1
> Host: POWERWALLGATEWAYIP
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json
> Content-Length: 96
>
* upload completely sent off: 96 out of 96 bytes
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Cache-Control: no-cache, no-store
Cache-Control: no-cache, no-store
* Added cookie AuthCookie="Or7INl<SNIP>aDQ==" for domain POWERWALLGATEWAYIP, path /, expire 0
< Set-Cookie: AuthCookie=Or7INlCz<SNIP>XPaDQ==; Path=/
Set-Cookie: AuthCookie=Or7INlC<SNIP>DQ==; Path=/
* Added cookie UserRecord="eyJlb<SNIP>wIn0=" for domain POWERWALLGATEWAYIP, path /, expire 0
< Set-Cookie: UserRecord=eyJlb<SNIP>AwIn0=; Path=/
Set-Cookie: UserRecord=eyJlbW<SNIP>wIn0=; Path=/
< Date: Fri, 02 Jul 2021 16:11:53 GMT
Date: Fri, 02 Jul 2021 16:11:53 GMT
< Content-Length: 260
Content-Length: 260
< Content-Type: text/plain; charset=utf-8
Content-Type: text/plain; charset=utf-8

<
* Connection #0 to host POWERWALLGATEWAYIP left intact
{"email":"MY@EMAIL.COM","firstname":"Tesla","lastname":"Energy","roles":["Home_Owner"],"token":"Or7I<SNIP>aDQ==","provider":"Basic","loginTime":"2021-07-03T02:11:53.028512297+10:00"}* Closing connection 0

Any further ideas?

Kind Regards
James

[jamesng]

If I drop the -k, the curl call fails in the same way as the Lua call

Is there any way to force my SHAC / lua to allow insecure server connections when using SSL like with cURL

Kind Regards
James

[admin]

HTTPS certificate verification is disabled by default. You can try sending a request using raw sockets to check where the remote host closes the connection:

Code:

IP = '192.168.1.9'
data = '{"username":"customer","password":"MYPASS","email":"MY@EMAIL.COM","force_sm_off":false}'

socket = require('socket')
ssl = require('ssl')

params = {
  mode = 'client',
  protocol = 'tlsv1_2',
  verify = 'none',
}

client = socket.tcp()
client:settimeout(1)

res, err = client:connect(IP, 443)
log('connect', res, err)

ctx = ssl.newcontext(params)
client = ssl.wrap(client, ctx)

client:settimeout(1)

res, err = client:dohandshake()
log('dohandshake', res, err)

res, err = client:send(
'POST /api/login/Basic HTTP/1.1' .. '\r\n' ..
'Host: ' .. IP .. '\r\n' ..
'User-Agent: curl/7.64.1' .. '\r\n' ..
'Accept: */*' .. '\r\n' ..
'Connection: close' .. '\r\n' ..
'Content-Type: application/json' .. '\r\n' ..
'Content-Length: ' .. #data .. '\r\n\r\n' ..
data)

log('send', res, err)

res, err, partial = client:receive('*a')
log('receive', res, err, partial)

client:close()

[jamesng]

That works and I can grab the AuthCookie as follows

res:match('(AuthCookie=[^;]+)'..';')

Many thanks!


My previous logic to retrieve info from the various API endpoints on the Powerwall also needs to be updated to pass the AuthCookie.

How can I use client : send to send the cooke as part of a GET request?

And also is there an easy way to return the result to a table to allow easy extraction of the returned data?


Here's the old code I'm trying to update

Code:

requestURL = 'https://POWERWALLGATEWAYIP/api/meters/aggregates'
requestData = ''
requestPostResponse = {}
token = GetUserParam('Wired', 'TeslaPowerwallLocalAccessToken')

res, err, commandPostHeaders, status = https.request
  {
    url = requestURL,
    method = 'GET',
    protocol = 'tlsv12',
    headers =
      {
        cookie = token,
        ['Accept'] = '*/*',
        ['Content-Type'] = 'application/json',
        ['Content-Length'] = requestData:len()
      },
    source = ltn12.source.string(requestURL),     
    sink = ltn12.sink.table(requestPostResponse)
  }
   
if err == 200 then
   
  mydata = json.pdecode(table.concat(requestPostResponse))
 
  teslaPowerwallPercentCharged = mydata.percentage
     
else
 
  log('Error retrieving Tesla Powerwall Battery Charge ' .. tostring(err))
 
  getTeslaPowerwallLocalAccessToken();
 
end

Many thanks

Kind Regards
James

[jamesng]

I've tried the following 

Code:

  socket = require('socket')
  ssl = require('ssl')
 
  client = socket.tcp()
  client:settimeout(1)

  ip = 'POWERWALLGATEWAYIP'
  token = GetUserParam('Wired', 'TeslaPowerwallLocalAccessToken')
  params = {mode='client', protocol='tlsv1_2',verify='none',}
 
 
  res, err = client:connect(ip, 443)
  ctx = ssl.newcontext(params)

  client = ssl.wrap(client, ctx)
  client:settimeout(1)
  res, err = client:dohandshake()

  res, err = client:send(
  'GET /api/system_status/soe HTTP/1.1' .. '\r\n' ..
  'Host: ' .. ip .. '\r\n' ..
  'Set-Cookie: ' .. token .. '\r\n' ..   
  'User-Agent: curl/7.64.1' .. '\r\n' ..
  'Accept: */*' .. '\r\n' ..
  'Connection: close' .. '\r\n\r\n'
   )

  res, err, partial = client:receive('*a')
  log('receive', res, err, partial)

  client:close()

however the way I'm passing the AuthCookie token doesn't seem correct as this is the error I get back from the Powerwall

Code:

* arg: 1
  * string: receive
* arg: 2
  * string: HTTP/1.1 403 Forbidden
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Mon, 05 Jul 2021 13:44:08 GMT
Content-Length: 102
Connection: close

{"code":403,"error":"Unable to GET to resource","message":"User does not have adequate access rights"}
* arg: 3
  * nil
* arg: 4
  * nil

How should the cookie / token value be set in the request header?

Kind Regards
James

[admin]

Replace "Set-Cookie" header with "Cookie"

[jamesng]

That works!

What's the best way to parse the response into a table so I can easily extract the values?

For example:

battery.instant_power
solar.instant_power
site.instant_power
load.instant_power

Here's a sample response

Code:

* string: HTTP / 1.1 200 OK
Cache - Control: no - cache, no - store
Cache - Control: no - cache, no - store
Content - Type: application / json
Date: Mon, 05 Jul 2021 14: 03: 05 GMT
Connection: close
Transfer - Encoding: chunked

8 b8 {
    "site": {
        "last_communication_time": "2021-07-06T00:03:05.470976626+10:00",
        "instant_power": -28.579999923706055,
        "instant_reactive_power": -1695.0799560546875,
        "instant_apparent_power": 1695.3208763576292,
        "frequency": 50,
        "energy_exported": 1540273.3280457454,
        "energy_imported": 40741163.06999019,
        "instant_average_voltage": 244.44000244140625,
        "instant_total_current": 0,
        "i_a_current": 0,
        "i_b_current": 0,
        "i_c_current": 0,
        "last_phase_voltage_communication_time": "0001-01-01T00:00:00Z",
        "last_phase_power_communication_time": "0001-01-01T00:00:00Z",
        "timeout": 1500000000,
        "num_meters_aggregated": 1
    },
    "battery": {
        "last_communication_time": "2021-07-06T00:03:05.490090497+10:00",
        "instant_power": 1450,
        "instant_reactive_power": 10,
        "instant_apparent_power": 1450.0344823486096,
        "frequency": 49.999,
        "energy_exported": 7754520,
        "energy_imported": 8834250,
        "instant_average_voltage": 244.20000000000002,
        "instant_total_current": -33.6,
        "i_a_current": 0,
        "i_b_current": 0,
        "i_c_current": 0,
        "last_phase_voltage_communication_time": "0001-01-01T00:00:00Z",
        "last_phase_power_communication_time": "0001-01-01T00:00:00Z",
        "timeout": 1500000000,
        "num_meters_aggregated": 2
    },
    "load": {
        "last_communication_time": "2021-07-06T00:03:05.469970299+10:00",
        "instant_power": 1461.8383691537629,
        "instant_reactive_power": -1656.2864201852196,
        "instant_apparent_power": 2209.1301734438607,
        "frequency": 50,
        "energy_exported": 0,
        "energy_imported": 36597478.81583333,
        "instant_average_voltage": 244.44000244140625,
        "instant_total_current": 5.980356547837028,
        "i_a_current": 0,
        "i_b_current": 0,
        "i_c_current": 0,
        "last_phase_voltage_communication_time": "0001-01-01T00:00:00Z",
        "last_phase_power_communication_time": "0001-01-01T00:00:00Z",
        "timeout": 1500000000
    },
    "solar": {
        "last_communication_time": "2021-07-06T00:03:05.469970299+10:00",
        "instant_power": 2.609999895095825,
        "instant_reactive_power": 57.65999984741211,
        "instant_apparent_power": 57.71904089514971,
        "frequency": 50,
        "energy_exported": 10814591.854183536,
        "energy_imported": 12338272.780294647,
        "instant_average_voltage": 244.47000122070312,
        "instant_total_current": 0,
        "i_a_current": 0,
        "i_b_current": 0,
        "i_c_current": 0,
        "last_phase_voltage_communication_time": "0001-01-01T00:00:00Z",
        "last_phase_power_communication_time": "0001-01-01T00:00:00Z",
        "timeout": 1500000000,
        "num_meters_aggregated": 1
    }
}
0

Many thanks

Kind Regards
James

[admin]

Try this, might need some adjustments to extract the JSON part from the response.

Code:

lines = res:trim():split('\r\n')

-- remove header lines and chunk size
repeat
  line = table.remove(lines, 1)
  if line == '' then
    table.remove(lines, 1)
    break
  end
until #lines == 0

-- remove last line
lines[ #lines ] = nil
data = table.concat(lines)
data = require('json').pdecode(data)

log(data.battery.instant_power)
log(data.solar.instant_power)
log(data.site.instant_power)
log(data.load.instant_power)

[mikeatlorne]

hey there, this appears to work for you and I hope it is still working. I am new to powerwall and am using a shac to control my house. I too want to be able to direct powerwall according to amber power price - which I have updating in my SHAC. what appears in this thread is therefore my missing link. However, I cannot get to first base. I've tried with "Set-Cookie" and "Cookie" and get the following error...

tesla trial 3 05.12.2021 15:04:56
* arg: 1
* string: receive
* arg: 2
* string: HTTP/1.1 403 Forbidden
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Sun, 05 Dec 2021 04:04:56 GMT
Content-Length: 102
Connection: close

{"code":403,"error":"Unable to GET to resource","message":"User does not have adequate access rights"}
* arg: 3
* nil
* arg: 4
* nil"

any ideas?

Mike