[Energy-Monitor] Energy Monitoring DIY Example

Writed: 14 Mar 20

이 글은 https://www.savjee.be/2019/07/Home-Energy-Monitor-ESP32-CT-Sensor-Emonlib/ 부터 복사해왔음을 알려드립니다.

 

Index

1. Introduce

2. Ingesting data

3. Archiving data

4. Calculating total consumption

5. Exposing data with GraphQL

 

1. Introduce

One day I was wondering: How much electricity am I using now? People don't know whether to turn on or turn off the air conditioner to save electricity, so they want to know the electricity bill now.

Looking online I found various sensor devices, but those are relatively expensive. So I decided to build my own.

1) Make a non-invasive energy monitor for the entire my home. Meaning no-cuttkng and not putting a meter bwteen every socket and light bulb.

2) Take measurements every second to get an accurate picture of electricity consumption.

3) Save all the data in the cloud for analytics

4) Have a simple app to visualize the data and analyze trends over time.

 

2. Ingesting data

I decided to make it and build myself a backend that can ingest this data. I'm quite an avid user of AWS and I have experience with their IoT service, Lambda, and DynamoDB. AWS was a no-brainer.

1) It has to be Serverless. I don't want to manage servers.

2) No fixed costs(pay for what you use)

3) It must be able to handle multiple sensors (you never know what the future brings)

4) Must be able to handle any data rate (ideally 1 reading every second)

5) Never throw away data! Always archive it and keep it for the future. You never know what might be intersting.

 

MQTT connection on AWS IoT service receive meesage and IoT Rule is treggered that writes the raw reading to a DynamoDB table. 

AWS users a message broker that dispatchers messages coming from your devices to one or more "listers"(eg a Lambda function). In this case, however, there aren't multiple consumers. Data should go straight into Dynamo.

 

3. Archiving data

Storing raw data in DynamoDB is great to be able to query the latest readings or to show an overview of the current day. It is , however, not the best long-term solution. The sensor is generating many data points every single day. That means that if I want to get an overview of how much electricity was consumed last month, I need DynamoDB to return at least many data. This would consume a lot of read capacity units(RCU). Normally DynamoDB offers 5 Read Capacity Units(RCU) and 5 Write Capacity Units(WCU).

I realized that older data doesn't have to stay stored in DyanmoDB. I cloud offload old data and store it on S3 instead! So, I made a Lambda function that is triggered at night and archives all readings from the past day to a single CSV file on S3.

 

This not only reduces my storage costs, but it also makes retrieving older data much more efficient. Getting all the readings from a particular day in the past is as simple as fetching 1 file from S3.

After setting up this Lambda function, I enabled DynamoDB's TTL feature. Time to Live(TTL) for Amazon DynamoDB lets you define when items in a table expire so that they can be automatically deleted from the database. With TTL enabled on a table expire so that they can be automatically deleted from the database. With TTL enabled on a table, you can set a timestamp for deletion on a per-item basis, allowing you to limit storage usage to only those records that are relavant. TTL is useful if you have contiunously accumlating data that loses relevance after a specific time period. If you have sensitive data that must be retained only for a certain amount of time according to contractual or regulatory obligations, TTL helps you ensure that it is removed promptly and as scheduled.

 

 

4. Calculating total consumption

This nightly lambda function also calculates the total amount of electricity(kWh) that was used during the previous day. This metric can then be used to visualize daily consumption over time. This calculated value is stored in DynamoDB so it can be quickly retrieved and doesn't have to be recalculated ever again.

 

5. Exposing data with GraphQL

I wrote a GraphQL API that runs on Lambda and lets me query the latest readings or readings between a given time range. I then wrote a simple web application(hosted in an S3 bucket) that visualizes the readings with Dygraphs. This was mainly for testing purposes until I had time to create an app. I broke out lonic framework, interfaced with the same GraphQL API.

 

Reference:

https://www.savjee.be/2019/07/Home-Energy-Monitor-ESP32-CT-Sensor-Emonlib/

DIY Home Energy Monitor: ESP32 + CT Sensors + Emonlib

https://graphql.org/

GraphQL: A query language for APIs.

http://dygraphs.com/

http://dygraphs.com/