In addition, historical data of your system is not enough. You would also need historical forecast data. Coming up with an optimal strategy if you know all the data is trivial 
The highest priority would not even be a simulator actually, that would be to create shared platform to bring all these individual DESS initiatives together in a structural way that allows us to compare functionality, performance, pro’s and con’s and above all a consistent way of defining a good requirements specification to tie it all together with. I’m not the best person for implementing the platform technicalities but I do have the systems engineering experience to look into the requirements specification and it’s subsequent derivatives (for a first impression look up v-model, not that that is the holy grail but it’s a start) Or check this: Volere Requirements Specification Template – Volere Requirements
The hardest part of creating a good solution is to come up with the correct problem definition, one that does not implicitly dictates any specific solutions upfront. That’s harder than you might think actually.
But first we would need a place to collaborate, could be on this forum, could be elsewhere, as long as it allows a structural improvement over all the ad hoc discussion scattered all over the place we are now having.
Thanks @bartm for highlighting the issue of large batteries draining over time. Turns out my assumption was wrong. A longer forecasting window will cause the battery to drain, only at a slower rate. This is unwanted behavior:
I overhauled the battery state of energy logic to be able to modify the objective function. The solver is now reimbursed for the energy that it has at the end. But I have found this behavior to be very difficult to steer using a min function or quantile(buy_price, 20%).
For this case a quantile(buy_price, 30%) + cycle_cost works:
But I imagine the current implementation of a quantile is very depended, on the number of cheap and expensive hours. What does seem to work is median(buy_price) * inverter_efficiency + cycle_cost, It captures the true replenishment cost, but I do not know how well it works in edge cases. For now I leave it, but maybe in the future a min SoE is needed to have the battery stay in a certain region for large systems.
Hi @nortciv. Nice simulations! Setting the terminal energy valuation to the minimum should have only an effect if there is excess solar. It will never buy energy from the grid to store in the battery because it’s not worth is (taking into account efficiency). If you increase it, the optimization should buy from grid at times where the buy price is lower than the chosen value.
During winter, when the risk of having a full battery due to large solar production is small, you could even argue that a fixed price makes sense: This should translate into “if the price is below x, buy from grid to use at a later time” behaviour. (note: in Belgium there are a lot of taxes, so a huge gap between buy and sell price. In practice thes means that there are almost no trading opportunities).
I’ve played around with creating a linear programming optimization myself and used a javascript solver (actually it’s webassembly). This allowed me to create a simple html front-end to play with the settings and get a feel for the effects of config changes. I’ve deployed it using github pages at https://optivolt.bartm.be/
I’ve loaded a simple demo config, but you can easily tweak the values. All changes will be stored in local storage in the browser. You can also share a config with someone using the share link. If I find time next weekend, I might try to fetch the predictions and prices from the Victron API.
1 Like
Wow, great work @bartm! I am absolutely stunned by how nicely your solution works (and looks)! The overall structure of both our solvers is largely the same (HiGHS, constraints, etc.). But the use of WASM is amazing! It plays much nicer with a possible future Node-RED solver block. I had thought about writing the solver in JavaScript myself, but my lack of JS programming experience forced me to prototype in python. Seeing how far OptiVolt is coming along makes me wonder if it would be more worthwhile for me to help you out sometime developing OptiVolt? I do have a few questions:
In the future, would it also be possible to create individual Node-RED dess building blocks? A LinearSolver block and a StrategyParser block could allow victron users to detach from the discontinued VRM Node-RED implementation while allowing them to build a custom DESS tailored to their need. I am not saying you should do this, I was just wondering if the use of WASM would allow for this?
Do you plan to implement some risk reducing logic in the solver? For example, I achieved this by taking prediction variance into account and view whatever solar is within the variance (error bars) as risky solar. In the objective function, I added the cost for this risky solar: self.price_sell + self.cycle_cost/2 + self.alpha_risk * (self.price_buy - self.price_sell. Alpha risk being between 0 and 1 just means: should risky solar be expensive or cheap. I am aware that this solution works best if the buy price and sell price are close to each other. If there is a large gap because of taxes, alpha needs to be really small to get the same behavior. Nevertheless, it helps users like me who have a small battery and way too much solar. But also users who have uncertain solar.
What size forecasting window are you hoping to work with? Keep it at 24 hours or more of a strategic multi-day planner? I can imagine that the ev-solver needs more than one day.
Anyway, If you want my help, just say so. I would gladly share my code/insights or invest some time in development/testing
Hi @nortciv
At the moment this is just a fun hobby project for me. I’m absolutely open to discussing ideas, exchanging code, point people in the right direction, etc. However, I’m not looking to make this a community project, manage it and provide support (I already have to do that at my job 
). For example, I want to have the freedom to make changes that might only be applicable to my installation.
Converting your Python code to JavaScript should be straightforward. The hard part is coming up with the equations and you already have those. Generating them in the right format is something that ChatGPT should be able to do. Especially if you already have Python code. In my repo this code can be found at optivolt/src/build-lp.js at main · bmesuere/optivolt · GitHub
I have zero experience with Node-RED, so I’m afraid I can’t answer that question. I have no idea if it allow running WASM code. I was thinking of running this locally using a lightweight webserver in a docker container myself. I would imagine that Node-RED has support for calling REST APIs.
Implementing a risk factor is something that might make sense for PV production. Alternatively, we could adjust the predictions based on the measured data of that day. Or just try to improve the predictions
I was also thinking about reducing the risk that occurs before a price peak when the actual load turned out to be higher than expected resulting in an empty battery and expensive grid usage. A mitigation might be taking the time of the morning and evening price peak, and setting the minimal soc 5-10% higher for the timestamp 30 minutes before the peak. This would allow for a bit of additional buffer.
For now, my biggest unknown is how to convert the optimal plan to something operational. Which API’s can we call? Use the VRM api? MQTT? Something else? Steering on expected SOC is not enough, you also need to set a strategy, but I don’t know if that’s possible.
HAve you guys seem ‘DAO’? This sound like something similar.
Won’t be long and DESS may be DOA, as in classical first-mover disadvantage (risk). Think Betamax, Nokia, Kodak. Open sourcing venusOS was a genius move, keeping the scheduler proprietary seems odd to me, same goes for the centralized storage of life performance data.
I’ve pushed an update that allows you to enter a token and fetch data from the VRM API (Everything is handled client side so your token is safe. There is only communication between your browser and Victron.) If you have DESS configured, it can autofill some settings and it also fetches forecasts and prices. The graphs were updated to work with actual times instead of generic time slots.
note: the victron forecast data is still hour-based. I split them over 4 equal15-minute intervals. For the PV data, it might make sense to do a smarter split.
The generated plan almost perfectly matches what I see in DESS, so it’s a good starting point for making tweaks to the rules if I/we spot situations where we think DESS should behave differently.
4 Likes
I just stumbled upon this branch by chance and find it extremely interesting. I’m not a programmer, but I’ve been an enthusiastic DESS user for about two years, through all the ups and downs. I think you’re currently working on one of the most critical aspects of the system: limiting the forecast period and dealing with energy surpluses at the end of the forecast period. These are two problems that I’m currently trying to resolve by adjusting the minimum Soc. Please continue working on this.
I think
and the entire development team should follow your work with great interest, if they haven’t already.
I read “Designed for integration with Home Assistant or custom dashboards” as if you were building a Home Assistant integration and publishing it that way. But looking back, I may have read it a bit too quickly. I also don’t want to be responsible for controlling other people’s systems in that way, that’s why I mentioned Node-RED.
If the solver could just be a node people can import, they could use it to build their own DESS. Node-RED runs on the Cerbo and has easy access to system parameters. The maintenance burden, forecasting implementation, and most importantly the responsibility would then lie with the end user. Plus, they could build in their own system-specific quirks.
Just thinking out loud, I’ll maybe look into it when I have some time next week. Anyway, I really like your implementation @bartm
I haven’t written any code yet to translate from the solver to strategies. However, here are some helpful sources: all the ModbusTCP registers are available (MQTT control is also possible and generally easier), and Victron’s own strategy interpretation is open-source and available in the file dynamicess.py
1 Like
Hi Bartm,
I’m getting a CORS error on the VRM API requests when trying with the token.
Hmm, I hadn’t noticed before because I used localhost myself. I just now added a small self-hosted proxy that forwards results and circumvents the error. I’ll later make the proxy configurable, so you can use your own.
1 Like
@nortciv Here’s the result from my research of how to turn the optimal plan into an operational strategy. I’ve pieced this together from forum posts, github repos and other sources so parts might be outdated.
First let’s understand how DESS works in VRM mode:
- Server side, an optimal plan is calculated from price data and expected load and PV over the window for which prices are known.
- The plan is converted into a strategy for 15 minute blocks and pushed to/pulled from the local device for the next 12 hours. (in practice only the first few quarter blocks are used because updated schedules become available).
- The plan for each window contains a number of variables (see below). In an ideal world, the plan sets a target SoC for the end of each window. From the current SoC and the target SoC you can calculate the needed charge/discharge rate and follow the plan perfectly.
- However, reality doesn’t follow predictions, so there must be a way to handle this. This is why locally, VenusOS determines a “reactive strategy” based on the planned strategy and actual data.
Next to VRM DESS, there is also Node-RED DESS. This also calculates the plan server side, but instead of pushing a strategy directly to our local devices, it is possible to make changes and set the strategy yourself. This happens using dbus registers under /Settings/DynamicEss/Schedule//…
- Start — Unix epoch (seconds), start of the window.
- Duration — seconds; typically 900 for 15 minutes.
- Soc — target state of charge (%) by the end of the window.
- AllowGridFeedin —
0/1toggle to permit export in this window. - Restrictions — behavioral hard limits (see below).
- Strategy — how to handle deviations (excess/missing energy vs the planned chargerate).
- Flags — underdocumented; I treat as 0 for now.
The possible restrictions are “no restriction”, “no battery to grid” and “no grid to battery”.
The possible strategies are “self-consumption”, “target soc”, “pro battery” and “pro grid”. These strategies define what happens if load or PV are different from the predictions. When using “self-consumption,” excess solar is pushed to the battery, excess load is taken from the battery. When using “target soc”, both are taken from the grid. In “pro battery” mode, the excess solar is pushed to the battery, excess load is taken from the grid. In “pro grid”, excess solar is pushed to the grid, excess load is taken from the battery. Note that from the descriptions on the forum, it seems that this is only about the excess PV and load and not necessarily for all load/PV. Although I think that in almost all cases this would behave identical since it’s based on the optimal plan.
Knowing this, I think it should be possible to
- put DESS in Node-RED mode to allow us to set strategies manually
- set the strategies using dbus/MQTT using our own code instead of Node-RED
This would let us piggyback on the reactive strategy stuff that’s already in place in VenusOS. We would then only have to find a way to convert our optimal plan into a combination of target SoC/restrictions/strategy settings. Target SoC is trivial since it’s an output of the LP. For the others, ideas are very welcome.
Note 1: There is a topic about deprecation of local DESS. In this post, Victron mentions pushing custom schedules to VRM. This would be a nice solution since that would mean we wouldn’t have to mess with dbus/MQTT. However, I don’t think such API exists yet.
Note 2: While the plan above has the advantage of reusing a lot of the available logic to come up with a reactive strategy, there is also the obvious downside that we would reuse any flaws in that logic. While some people are very vocal about these, to me this only a minor inconvenience compared to the possibility of using a custom LP program that incorporates a heat pump schedule, EV charging and other tweaks. Since the only alternative is to have a script constantly updating the grid setpoint, I would propose to keep this discussion out of this topic and focus on how to come up with sensible restrictions/strategies to put in the schedule.
Thank you for your extended summary @bartm
I would really really like to prevent a reversal to DESS mode 4 a.k.a. Node-RED DESS. Seems that the key question moving forward concerns the VRM-API modify function of which I (for one) cannot really find any good documentation or even any examples. What I can say for certain is that “/Settings/DynamicEss/Schedule/” D-Bus registers work just as well in DESS mode 1 (VRM DESS), I actively use the SoC and Strategy registers to nudge the DESS execution engine in the right direction.
First by reading and filtering for change (up to 8 times per hour, all nodes set to “on change”)
And then use that “change” signal to immediately push any modified SoC / Strategy values to the correct register (for current timeslot, see dbsindex) to be picked up by the DESS execution process. And therefore have the modified values instantly forwarded to the “Set Target SoC% for this timeslot” and “Used Strategy for current time slot” that drive the inverter.
For instance: in the flows above, dbsindex will activate what PIN will be used on those switches (in red) and whenever the reactive strategy becomes “9: IDLE MAINTAIN TARGET SOC”, this will automatically push strategy “1: Self Consumption” to the correct d-bus register to be picked up by the “Used Strategy for current time slot” within about 5 seconds. And the reactive strategy to “1: SCHEDULED SELFCONSUME” as well.
All while running DESS mode 1 that is. And yes I know all that can be implemented in a single or just a few function nodes but I like the visual clarity that using (boolean logic ultimate) specialty logic nodes provides, such as these “railway switch” nodes above.
Here the code for that “ReactiveStrategy” function node. It doesn’t do anything special but providing human readable output to the debug node.
var mapping = {
1: 'SCHEDULED_SELFCONSUME',
2: 'SCHEDULED_CHARGE_ALLOW_GRID',
3: 'SCHEDULED_CHARGE_ENHANCED',
4: 'SELFCONSUME_ACCEPT_CHARGE',
5: 'IDLE_SCHEDULED_FEEDIN',
6: 'SCHEDULED_DISCHARGE',
7: 'SELFCONSUME_ACCEPT_DISCHARGE',
8: 'IDLE_MAINTAIN_SURPLUS',
9: 'IDLE_MAINTAIN_TARGETSOC',
10: 'SCHEDULED_CHARGE_SMOOTH_TRANSITION',
11: 'SCHEDULED_CHARGE_FEEDIN',
12: 'SCHEDULED_CHARGE_NO_GRID',
13: 'SCHEDULED_MINIMUM_DISCHARGE',
14: 'SELFCONSUME_NO_GRID',
15: 'IDLE_NO_OPPORTUNITY',
16: 'UNSCHEDULED_CHARGE_CATCHUP_TARGETSOC',
17: 'SELFCONSUME_INCREASED_DISCHARGE',
18: 'KEEP_BATTERY_CHARGED',
19: 'SCHEDULED_DISCHARGE_SMOOTH_TRANSITION',
20: 'SELF_CONSUME_ACCEPT_BELOW_TSOC',
21: 'IDLE_NO_DISCHARGE_OPPORTUNITY',
92: 'DESS_DISABLED',
93: 'SELFCONSUME_UNEXPECTED_EXCEPTION',
94: 'SELFCONSUME_FAULTY_CHARGERATE',
95: 'UNKNOWN_OPERATING_MODE',
96: 'ESS_LOW_SOC',
97: 'SELFCONSUME_UNMAPPED_STATE',
98: 'SELFCONSUME_UNPREDICTED',
99: 'NO_WINDOW'
};
//msg.payload = msg.payload + ': ' + ( mapping[msg.payload] || 'Unknown' );
msg.reactivestrategy = msg.payload + ': ' + ( mapping[msg.payload] || 'Unknown' );
return msg;
Can you explain why you are against using “mode 4”? Do you lose other things in mode 4? I’m not seeing much use in letting VRM set the schedule which we would then override anyway? Quite the opposite, we would have to make sure that VRM doesn’t overwrite our own schedule in mode 1.
1 Like
Seamless VRM integration for future customers.
A. All you need for that can be done with the flows I showed above. The explicit function of the top one is to instantly ‘capture the moment’ VRM pushes any schedule changes to venusOS, allowing us to block and/or adjust changes to the D-Bus registers of interest. Do you want a private copy?
B. We should discuss with Victron how we can provide the correct pre-scheduling input to their cloud based DESS scheduler, to calculate the required schedules we need in the first place. I am of the firm conviction that all I need for that is a port of the Node-RED DESS schedulers b_goal_SoC/b_goal_hour function to VRM DESS. Because that will allow us to set (at least one) future SoC setpoint for the scheduler to target, before pushing its results onto the VenusOS DESS execution engine. And I can know because I have seen this work perfectly in Node-RED DESS, when we still had hourly price schedules. That will provide a more than adequate general direction day ahead schedule, any by the hour or less timescale deviations thereupon can be done locally, And will then be picked up every hour by the cloud scheduler for day ahead corrections.
Aha, I think I found the misunderstanding. I don’t really care for NodeRED and do not plan to use it. I just want to push values to MQTT/dbus from my code. I would only set DESS to mode 4 to signal that I want to push the schedules myself. As I noted, doing this using the VRM API instead of dbus would be preferable, but for now mode 4 seems the pragmatic approach.
Fair enough, you have no customers I presume? For private use I’d fully agree.