HomeStation
Design and construction of a solution for the measurement and storage of environmental parameters in the home environment using ESP32, C++, ESP-IDF, MQTT, ASP.NET and containerisation.
Introduction
My purpose in creating this was to focus on connecting different environments and tools together to provide a solution to user to measure temperature, humidity, pressure and particulate matters (PM) levels.
Of course it didn’t happen overnight and I had to quickly learn quickly many technologies just to run basic functionality (i.e connect device to Wi-Fi, subscribe and send hello world to mqtt broker). The concept changed a few times, but the maintain idea stayed intact.
Used technologies
Let’s be clear. Everything wasn’t planned by obsessing over a lot of models and graphs. I had an initial concept in my mind:
- I need a device with a sensor that logs data
- I need to receive data
- I need to store data
- And finally I need to show data.
So, knowing the .NET environment and a little bit of Angular, I decided that they would be my go-to in this project.
Much worse was the device. .NET nanoFramework, Python, Rust, C, C++? What should I choose? After looking at and toying with a few (and messing with one for a bit longer), I’ve decided to go with C/C++ with ESP.IDF (sorry Arduino).
At this point I know which tools to use, but there are still two questions - where and how to store the data, and how to send this data from the device?
The first question was easy to answer - MSSQL. Of course I could use the MySQL, Postgre, Mongo, etc., but first - I know MSSQL better than any other solution in this matter. Secondly, I don’t need to use NoSQL, because the data is structured and this structure is immutable, so storing it as e.g JSON is a bit of overkill. Finally, there is a pre-built container image.
The second question, was not simple as the first one. Should I use REST API? Well ESP32 with ESP-IDF onboard can send POST requests, but what if I want to collect measurements from more than one device? For example - two, five, hundred, thousand? As you can see as the number of devices increases the problem grows, due to sending large amount of POST requests being sent. After googling and looking back and forth I stumbled upon MQTT, and there is a use-case describing the same situation as mine. I didn’t want to add another layer just for maintenance (but it could be done), so i skipped solutions like mosquitto. In the end I came across MQTTnet, which could be implemented alongside my REST API.
The final tech stack should look like this:
- ASP.Net - I wanted to communicate with the UI somehow, REST API seemed the best,
- Entity Framework - ORM just to store/read data to/from database,
- MQTTnet - For device to subscribe and send message under topic
- ESP-IDF - IoT framework for device to communicate with sensors and a broker
- Angular + Angular Material - Framework and set of already prepared controls for the UI
- Chart.js - Data needs to be presented as graph
Device
I knew exactly which environmental parameters I wanted to measure. I had two DHT22s in my drawer, but they didn’t work at all. I browsed the shops and ordered a Bosch BME280 (which will cause problems in the future), and a Plantower PMS3003. I also added HD44780. The last thing left was the board, which ESP should I choose. I already had two Wemos D1 R32, but they’re are quite big, so after some consideration I decided to buy ESP32-WROOM.
Physical connection
The final connection went as follows. Everything is soldered on the prototyping board with longer cables to the BME280 to avoid heat from the ESP32 or other devices.
Overall I had to change my original connections due to that some of the pins are reserved and connecting to them causes that the device not to boot up.
Code
What I’ve googled the best solution was to assign MQTT client and PMS client to separate tasks. BME280 and printing data to HD44780 on the main method.
The reason to staying on main was to less troubles with interrupts. From my observation when I am communicate with device on raw GPIO port if interrupt happens I will miss timings in milliseconds that are required by hardware.
3D Printing
Our cable-soldering mess needs protection if devil creatures decide to destroy it. My options were limited for me, because I own a 3d printer, so I went for it.
A hour or two in blender later and It’s done. Time to print.
Backend and Frontend
As I mentioned in Used technologies, I wanted to receive and store data. The device connects via WiFi and tries to subscribe to non existing MQTT broker. This means that some kind of a broker, database is needed to process further. Of course raw readings mean nothing, if I can’t present them in an easily readable form.
Backend
At this point, I decided to play it safe and went with the known clean architecture (but it’s a bit overkill). Quickly created projects in my solution, and after reading MQTTnet documentation, deployed the broker.
Eureka! I’ve connected to the MQTT broker. Time to flash device with specific connection parameters and try to send data.
The readings have been sent and received via the API. So now all that’s left on this side is to put it into database.
Quickly prepared options pattern for the database’s required values, and prepared classes to create tables from them (code-first approach).
The database looks like this:
I added database migration and applied it to a local instance of Microsoft SQL Server.
After running API again:
At the moment, the device was communicating with API through this interface
{
"deviceid": 1 //number,
"temperature": 24.9 //double,
"humidity": 44 //double,
"pressure": 100321.6 //double,
"pm1_0": "5" //number,
"pm2_5": "9" //number,
"pm10": "9" //number
}
As you can see the read date property is missing here. The reason behind this approach is to avoid maintaning the clock as physical part of the device (the clock has to be part of the device with an additional power source from battery to ‘hold’ the date and time between reboots), or calling the NTP provider periodically/every time to send data.
Frontend
The data can be stored and received as JSON, which isn’t very pleasant to read it when there is a large amount of data.
The UI was a must from the start. After a while I added a sidebar with device management, a page with current readings, and tabs with graphs.
The data could be displayed now, but what if the user wants to look at the long-term data? Not a day or two, but say - ten or even a hundred?
It’s worth using zoom plugin for chart.js, which solves this problem.
After a few days of testing and tweaking the code it resulted in this layout:
Containerisation
After all I could call it a day, but I didn’t know which system or even cloud I was going to use to deploy it. So, there are two answers, either I going to build and test on every system possible or use containerisation. I tend to stick to the second options, due to effiency. This time I prepared files for Docker containerisation platform and Kubernetes for running and managing containers.
Docker
There are an two docker files. One for the UI and one for the backend. Resource-intensive containers weren’t what i wanted, so UI is uses nginx, and backend aspnet:8.0 (it can be changed to alpine). Both dockerfiles are multi-stage - I’ve created them to prepare environment, build and deploy. A one-click solution to deploy needed, so both files could be run from docker compose.
volumes:
sqlserver_data:
networks:
homestation:
driver: bridge
services:
homestation_db:
container_name: homestationDb
image: mcr.microsoft.com/mssql/server:2022-latest
environment:
- SA_PASSWORD=<AwesomePassword> #Worth to mention that you can use secret
- ACCEPT_EULA=Y
ports:
- "1433:1433" #Database port
networks:
- homestation
volumes:
- sqlserver_data:/var/opt/mssql
restart: always
homestation_api:
container_name: homestationApi
environment:
- ASPNETCORE_HTTP_PORTS=80
- Database__ConnectionString=Data Source=<AwesomeIp>,1433;Database=homestation;User Id=<AwesomeLogin>;Password=<AwesomePassword>;Encrypt=False;TrustServerCertificate=True #Also here
build:
context: ./Web
dockerfile: Dockerfile
ports:
- "1883:1883" #mqtt port
- "9180:80" #api http port
networks:
- homestation
depends_on:
homestation_db:
condition: service_started #I want to wait until database service started
restart: always
homestation_web:
container_name: homestationWeb
build:
context: ./Web/web.client
args:
- HREF=/homestation/ #could be changed to '/', the UI is predicted to use prefix /homestation/ - i.e localhost:9080/homestation/
ports:
- "9080:80" #http UI
- "9443:443" #https - just in case
depends_on:
- homestation_api
networks:
- homestation
restart: always
After a few tweaks the whole solution can be deployed with docker compose -f compose.yaml up -d
I urged to be fancy - Kubernetes
At this point the project is ready to run on a containerisation platform, but I wanted to go further. So i set up the k3s kubernetes distribution, and started developing yaml files. After much trial and error, and installing ingress-nginx via helm (yes, on k3s you have to disable traefik, if you want to use ingress-nginx like I do), I’ve got it working on my kubernetes instance.
So let’s dive in:
- Created logal image registry
- Uploaded image into registry
- Created namespace -
kubectl create namespace homestation - Added secrets:
#secrets have to be in base64 apiVersion: v1 kind: Secret metadata: name: homestation-secrets namespace: homestation data: #Data Source=<AwesomeIp>,1433;Database=homestation;User Id=<AwesomeLogin>;Password=<AwesomePassword>;Encrypt=False; ConnectionString: RGF0YSBTb3VyY2U9PEF3ZXNvbWVJcD4sMTQzMztEYXRhYmFzZT1ob21lc3RhdGlvbjtVc2VyIElkPTxBd2Vzb21lTG9naW4-O1Bhc3N3b3JkPTxBd2Vzb21lUGFzc3dvcmQ-O0VuY3J5cHQ9RmFsc2U7 #P@ssw0rd DbPassword: UEBzc3cwcmQ= - Deployed database server:
apiVersion: v1 kind: PersistentVolume metadata: name: sqlserver-data namespace: homestation spec: accessModes: - ReadWriteOnce capacity: storage: 5Gi hostPath: path: "/var/opt/mssql" --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: sqlserver-claim namespace: homestation spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi --- apiVersion: apps/v1 kind: Deployment metadata: annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) labels: app.kubernetes.io/name: homestationdb-deployment name: homestationdb-deployment namespace: homestation spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: homestationdb strategy: type: Recreate template: metadata: annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) labels: app.kubernetes.io/name: homestationdb spec: containers: - name: homestationdb image: mcr.microsoft.com/mssql/rhel/server:2022-latest ports: - name: "1433port" containerPort: 1433 - name: "1434port" containerPort: 1434 env: - name: ACCEPT_EULA value: "Y" - name: SA_PASSWORD valueFrom: secretKeyRef: key: DbPassword name: homestation-secrets volumeMounts: - name: sqlserver-data mountPath: "/var/opt/mssql" volumes: - name: sqlserver-data persistentVolumeClaim: claimName: sqlserver-claim restartPolicy: Always --- apiVersion: v1 kind: Service metadata: annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) labels: app.kubernetes.io/name: homestationdb name: homestationdb namespace: homestation spec: ports: - name: "1433port" port: 1433 targetPort: 1433 protocol: TCP - name: "1434port" port: 1434 targetPort: 1434 protocol: UDP selector: app.kubernetes.io/name: homestationdb - Created database:
apiVersion: batch/v1 kind: Job metadata: name: homestationdb-prepare namespace: homestation spec: template: spec: containers: - name: homestationdb-prepare image: mcr.microsoft.com/mssql-tools command: ["/opt/mssql-tools/bin/sqlcmd"] env: - name: DbPassword valueFrom: secretKeyRef: key: DbPassword name: homestation-secrets args: [ "-S", "homestationDb", "-U", "sa", "-P", "$(DbPassword)", "-C", "-I", "-Q", "IF NOT EXIST (SELECT * FROM sys.databases WHERE name = 'homestation') BEGIN CREATE DATABASE homestation; END;" ] restartPolicy: Never backoffLimit: 4 - Deployed backend:
apiVersion: apps/v1 kind: Deployment metadata: annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) name: homestationapi-deployment namespace: homestation spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: homestationapi template: metadata: labels: app.kubernetes.io/name: homestationapi spec: containers: - env: - name: MQTT__Port value: "1883" - name: MQTT__Address value: "0.0.0.0" - name: ASPNETCORE_HTTP_PORTS value: "80" - name: Database__ConnectionString valueFrom: secretKeyRef: key: ConnectionString name: homestation-secrets image: 127.0.0.1:5000/homestation2-homestation_api:latest #specify image registry name: homestationapi ports: - containerPort: 9883 protocol: TCP - containerPort: 9180 protocol: TCP restartPolicy: Always --- apiVersion: v1 kind: Service metadata: labels: app.kubernetes.io/name: homestationapi name: homestationapi namespace: homestation spec: ports: - name: "9883" port: 9883 protocol: TCP targetPort: 1883 - name: "80" port: 9180 protocol: TCP targetPort: 80 selector: app.kubernetes.io/name: homestationapi --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: homestationapiingress namespace: homestation annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /api/$1 spec: rules: - http: paths: - path: /api/(.*) backend: service: name: homestationapi port: number: 9180 pathType: ImplementationSpecific ingressClassName: nginx - Deployed Frontend:
apiVersion: apps/v1 kind: Deployment metadata: namespace: homestation annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) name: homestationweb-deployment spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: homestationweb template: metadata: annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) labels: app.kubernetes.io/name: homestationweb spec: containers: - image: 127.0.0.1:5000/homestation2-homestation_web:latest #specify registry name: homestationweb env: - name: TARGET_URL value: "http://homestationApi/homestation/" ports: - name: "https" containerPort: 9443 - name: "http" containerPort: 9080 restartPolicy: Always --- apiVersion: v1 kind: Service metadata: namespace: homestation annotations: kompose.cmd: kompose -f compose.yaml convert kompose.version: 1.35.0 (9532ceef3) labels: io.kompose.service: homestationweb name: homestationweb spec: ports: - name: "http" port: 9080 protocol: TCP targetPort: 80 - name: "https" port: 9443 protocol: TCP targetPort: 443 selector: app.kubernetes.io/name: homestationweb --- apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: homestationwebingress namespace: homestation annotations: nginx.ingress.kubernetes.io/use-regex: "true" nginx.ingress.kubernetes.io/rewrite-target: /homestation/ spec: rules: - http: paths: - path: /homestation/ backend: service: name: homestationweb port: number: 9080 pathType: Prefix ingressClassName: nginx
Then, after executing kubectl get pods -n homestation I’ve received this:
Summary?
Overall, the project expanded the knowledge of IoT devices, how they work and how to build software on microcontrollers.
Notable is kubernetes, which stopped being a ‘black-box that works thanks to CI/CD’, and started being a container management platform with services, deployments, ingresses etc.
Lastly, 3D printing - finally I have the opportunity to use it in a real project, not only to print useless stuff that wil end up in a drawer for years.
Mistakes
BME280 heating
You should be aware that this sensor is self-heating to provide accurate humidity and/or pressure readings, so consider setting oversampling to 1X, and using it through force mode. If you can consider different sensor.
Try to being modular
I’ve tried putting goldpins everywhere to quickly replace broken parts, but you have to know that there are many manufacturers of these, and they vary in quality. In particular, it quickly became annoying when the HD44780 display sometimes disconnected.
No cache
If I select about 30 days in the UI with the detailed view and try to download data, I’m going to wait for a while (and even longer if my computer couldn’t render all the points). It’s good to have a proxy between the UI and the backend.
.Include().ThenInclude().Include()
SELECT is so much faster than doing multiple includes. Entity framework translated them to many left joins, and they cause terrible performance.
Appendices
HD44780
After some time of use I’ve disconnected the display, and removed code from repository. Now I’m reading data from the web. The reason is simple - poor quality of my goldpins, and I want to avoid soldering prototype board again. There is last working version with display - click