Disclaimer: This can be irrelevant, but I thought I’d share. I am doing an experiment, so it’s all theoretical and work in progress. If you see possible improvements or obvious errors in my reasoning, please point them out!
I have been investigating if I can develop my own DESS algorithm using linear programming. I have no clue what techniques Victron is using to solve their schedules, but the current implementation has some limitations. For example:
My system regularly overshoots the target SOC and then discharges if the schedule is not updated in time. A complete waste of energy.
Inaccurate solar forecasting, or irregular consumption for that matter, throws off the DESS, and it has to buy expensive energy. In my case, the time of consumption for my EV is known (thanks to EVCC.io).
This LP algorithm minimizes total energy cost over any time horizon using consumption and solar forecasts, along with system parameters. The solver accounts for constraints such as battery capacity, maximum charge/discharge rates, conversion efficiency, cycle cost, and grid limits. For a 24-hour window, it solves in 1.78 ms ± 47.1 μs on my laptop. Though I do not plan to deploy it on the Cerbo, it will run on a dedicated server and be called frequently to adapt to changing consumption or low solar yields.
I still need to work out the details to control the Victron system with ModbusTCP and get the forecasting models. Any ideas?
The end goal is to hopefully implement sensitivity analysis to the solver. Some kind of system that could balance an optimal schedule with forecasting uncertainties. Also, this system can be made more robust with more auxiliary variables. It could automatically plan EV charging or a heat pump if they allow for external control. It should also be possible to let it schedule a generator. But that is all future talk. It does add to the complexity and would add to the solve time.
After further experimentation, the system appears quite promising. I’ve made additional tweaks to the solver to more accurately reflect the State of Energy (SoE) rather than the State of Charge (SoC). This means system efficiency is now directly embedded in the SoE, instead of being indirectly compensated by altering the price. The next step is sensitivity analysis—adjusting forecasts to account for varying levels of uncertainty and observing how the optimal solution (the SoE schedule) responds. Initially, only the solar forecast is modified, with uncertainty arbitrarily increasing over time.
The optimal solution depends on the available solar, but only to a certain extent—this sensitivity is influenced by the price. Even in scenarios where no solar generation is forecasted, the system eventually reaches a point where it begins charging from the grid.
In between these two runs are a lot of solutions where the systems adjust the time of charge to accommodate the available solar energy. The sensitivity analysis graph can be seen as a min/max graph for the systems SoE.
Next up
Next steps include translating the sensitivity analysis into actionable grid strategies compatible with Victron. Such as SoC-level control, pro-battery, and pro-grid modes. It may also be necessary to extend the sensitivity analysis to consumption forecasts for better strategy decision. Simulations will follow where the algorithm is “surprised” by a downward adjustment in the solar forecast at hour x. The goal is to observe how this impacts the solution and whether tuning the uncertainty parameters can mitigate the disruption. If successful, the system will be deployed on a test setup.
Should the experiment prove effective, I plan to open-source it for the community. Possibly as a Home Assistant plugin, a Node-RED implementation or a external docker image. I don’t know yet.
I’m also working on such an optimization thing, I got it to the stage where it controlled my ESS and evcc, but then the 15 minute pricing came and I haven’t yet found the time to fix the code to work with those.
The best way I found to control the ESS is by using the DVCC CCL if charging is to be less than the available PV surplus, setting the minimum SOC to the current SOC to basically shut off the inverter, or setting a grid setpoint for discharging to the grid:
def set_victron(self, minsoc, CCL, grid_sp):
if CCL is None: CCL = 65535
if grid_sp is None: grid_sp = self.grid_sp
client = ModbusTcpClient(self.cfg.get('API', 'venus_host'))
client.connect()
client.write_register(2901, int(minsoc*10), device_id=100)
client.write_register(2705, int(CCL), device_id=100)
client.write_register(2700, int(grid_sp) & 0xffff, device_id=100)
client.close()
evcc is easier with its REST API.
I then did a few tweaks to the inputs, such as adding a tiny fraction of the EPEX spot price to my fixed sell price to nudge the solver towards exporting during the price peaks (i.e. in the morning) and delaying charging until later, which worked pretty well, and adding another tiny increment towards the end of the horizon, to prefer charing sooner when forecasts are more accurate and not delay it until the future where there might be clouds.
But it mostly did what evcc alone would have done, except for one dark rainy week where my battery ran so low it decided to disable the inverter during the (cheap) night hours and run from the grid instead – exactely what I was expecting.
Thank you so much for sharing your insights! Especially EpexPredicter, I am in the midst of training my own forecasting models. I will investigate how I can use EpexPredictor, because I also use evcc.
Also is there a reason you chose the DVCC registers instead of the new DynamicEss registers? I have not implemented them yet, but on first glance they seem very flexible. You can set a Soc, UNIX start time and Strategy:
/Settings/DynamicEss/Schedule/0/Soc in %
/Settings/DynamicEss/Schedule/0/Start time when schedule should start
/Settings/DynamicEss/Schedule/0/Strategy (0=Target SOC;1=Self-consumption;2=Pro battery;3=Pro grid):
Here is how I made the solver, maybe it is of any use:
I used the scipy linprog solver which allows me to plug in any time horizon and it will solve for that. If it is minutes, 15 minutes, hours or days, the solver does not know nor care. As long as all the input arrays are the same size it will work. Another advantage is that I can let it calculate partial days. For example, I can adjust the forecasts during the day and remove the hours that have already passed. The solver will now find the optimal strategy for the next hours with the new forecasts. But from your post I guess you are using something similar.
Instead of tweaking the price, the solver is run multiple times with less solar available (based on solar forecasting error). Late hours are reduced more than early hours. The profits are compared between strategies and the strategy that is within €0.05 of the optimal strategy is chosen. This ensures a balance between risk reduction and maximizing profits.
I have DESS turned off, can I still use those registers? And do I really want to prescribe a SOC? What if there’s more, or less, sun than predicted?
I’m using optlang, mainly because I got access to some similiar code using it and learned from that (it’s also my first real Python project, I’m normally a Perl person, but there’s simply no match for the combo of pandas + optlang + plotly in the Perl world).
And yes, my solution’s also flexible WRT time horizon, but my code is currently heavily based on “1 kWh in 1 hour = 1 kW” and I still have to go thru it and do the conversion from and to quarter hour (or whatever) intervals, and I’m currently busy pre-wiring for more solar panels and mounting an MPPT and so on…
One nice thing about pandas is that you can just throw series with differing intervals into it and have it do some magic and convert them all to the same interval – and I’ve also been running the code from cron every hour and then just apply the result for the current hour, no need to store the plan for the whole day as calculations only take a few seconds even on a RK3399 ARM board.
I should probably get that thing finished (ha!) and presentable and publish it somewhere.
PS: my “consumption forecast” is simply “200W constant”, which matches my patterns well enough. I’ve experimented with using the forecasts from the VRM API, but they are all over the place because the EV charging is in there, but what I need is just “the house”. My bigger loads, dishwasher and washing machine, are on such an irregular schedule that forecasting makes less sense than just letting them consume energy that will be taken into account on the next optimizer run. But I also have a 28kWh battery…
Following with interest. Will join when I can find time to refactor our own hybrid DESS scheduler. (Forum search for details, project got stuck for lack of Victron response to engage in constructive dialog concerning two mayor flaws in DESS Trade:
Lack of precision/resolution for the DESS quarter hourly scheduled SoC values (now INT values, resulting in unsolvable aliasing issues with large capacity batteries where 1% SoC per 15 minutes (of energy) is in the same order of magnitude as the maximum systems power performance to charge / discharge that same amount of energy.
Undisclosed/hidden assumptions behind the DESS (Trade) scheduler that there will be ( sufficient / any ) PV power installed to recharge the battery the next scheduling day. Which for a Trade only (no solar at all) system results in a massive sell bias to always trade towards minimum SoC at the end of the known price schedule, despite obvious and predictably profitable market opportunities being present, even within the limited timeframe of known prices. The Node-RED DESS scheduler allows somewhat of a workaround by enabling a programmable buy bias towards a target SoC at a target hour, as an input to the scheduler . Unfortunately and to my dismay, any and all attempts to convince Victron to port this feature to VRM DESS have stranded on hopeless discussions about DESS Green mode and DESS settings without ever acknowledging the actual root cause of problem. Our hybrid DESS system ran both schedulers in parralel to overcome this regression in VRM DESS but stopped working with the introduction of 15 minute pricing by our energy supplier. Hence my remark about refactoring.
Hi Jan, thanks for chipping in. Just for my clarity isn’t the SoC precision also a BMS limitation? Some systems just have a BMS that publishes integer values. The precision is also why I refrain from using the SoC instead I use SoE in all my calculations (only go back to SoC if I need to).
We use BMV Smartshunt data to calculate SoC as 1 - (Consumed Ah / Capacity Ah). That provides 0.005% precision (0.1Ah/2048Ah) in our case. And I am building a SoE counter as well just to remove the non linearity otiginating from the variable voltage range. The aliasing effect relates to the fact that a 88kWh battery with a 4.4kW inverter can sell 5% capacity per hour, 1.25% per 15 minutes. The schedules are always wrong
-1 -1 -1 -1 = -4 didn’t sell -5%
-2 -1 -1 -1 = ? Can’t sell -2%
-1 -1 -1 -2 = ? Sold -3% can’t sell -2%
The only way to prevent throttling during the hour is to ‘pretend’ the inverter can deliver over 6kW which only pushes the problem to the scheduler to correct the missed sale every hour. Lack of resolution cannot be resolved by adding correction layers, this is an aliasing effect albeit with INT SoC% datapoints instead of pixels / waveforms. Scheduling overshoots and undershoots are a scientific given when the smallest energy target unit per timeframe is in the same order of magnitude as the largest energy (pump/transport) performance for that same timeframe. I am pretty stunned I can’t seem to find anybody within the dev team that understands this to be a mathematical fact.
Reading with interest, @nortciv! I have toyed a little with my own logic, borne from frustration with the tyre fire that is the current state of DESS and the lack of engagement from Victron.
I don’t have much to share just yet, only tinkerings and half-baked code: your implementation is much further down the track than mine. Bravo! It seems like you’re thinking along the same lines that I was, only you’ve had much more intelligent thoughts
So, this message is mostly one of support and encouragement. I wish I had some more time to devote to this, but alas, I am very time poor for a while longer yet. I will watch your work with interest!
Hi Andrew thanks for showing interest . This experiment is also only worked on when I have a bit of spare time. But I have a small update to share. After having a chat with an expert in the field of linear programming I moved away from sensitivity analysis and added an integrated risk function with a risk taking parameter called alpha.
With an alpha of 0 the solver ignores any uncertainties in the solar forecast, it focuses on price only:
The idea is that you tune alpha to your liking. Because alpha is integrated in the cost function it takes the price difference into account. A small alpha will quickly move charges to different hours if prices are close to each other, but will not change hours if one hour is particularly cheap (or others expensive). This is the balance I wanted when I said: “balance risk with profit”.
I hope to have some time upcoming days to work on the in-hour strategy and interface to the Victron equipment.
I’m also very interested in your work! I’m a computer science professor myself, so if you need a hand I’m happy to help.
An idea I had earlier is to improve the planning by increasing the forecast window. The Victron algorithm behaves like the world ends when no price data is available and thus always optimizes to reach an empty battery at the end of the window. In some cases, this forces a discharge when it doesn’t make sense. It seems relatively easy to predict an estimate a few days in advance which would allow to create a better planning.
Hello, I am also following this development with interest.
I myself have used AI to build a simple battery charging and discharging system into the distribution network in Nodered in Cerbo. I only take the current 15-minute interval during the day from the API provided by https://spotovaelektrina.cz/. Depending on the season, I set how many 15-minute intervals the battery should be charged to 80% SOC from the distribution network. Once a week, I charge it to 100%.
For discharging, I have set only discharging according to the current price per quarter hour for the winter period. When above CZK 8, discharge to 70% SOC in winter. In summer, the price is CZK 4 and the SOC limit is 50%.
I want to intervene dynamically according to the production forecast from open.meteo.
Another possible development for me is predicting household consumption. This mainly concerns winter, when I heat using a heat pump. I am already collecting data on its consumption and the current outdoor temperature. So hopefully the consumption forecast will be useful.
I’m keeping my fingers crossed for the project.
Have a nice and sunny day from the Czech Republic.
In the lexicon of these things, a “hot mess” is a shorthand for a variety of types of Scheißeshow, covering tire fires, boiling garbage, train wrecks, garbage fires and goat rodeos
I should add: I say this in both good humour and frustration. This forum is largely comprised of Victron fanboys/girls who spend lots of money on Victron products and would love to see this feature work better. This thread is testament to the incredible amount of knowledge, skill and effort the community has to contribute. Imagine what could be achieved if Victron embraced that!
What do you do with the energy in the battery at the end of the forecasting window? I think it makes sense to take it into account into the objective function, for example by subtracting the “replacement cost”. This could be SoE*min(import cost) as a conservative estimate.
What do you use as your source for load and PV predictions? Still what Victron provides? Or do you have your own model? It would make sense to keep EV charging out of the load prediction.
Do you plan to keep using EVCC? For planned charging, it should be possible to convert those constraints into the linear program. The downside is that the program would need some UI to define the input instead of just running headless on a server. You could even allow to set time ranges when you know the car won’t be at home by restricting the bounds for those time slots.
Have you already found a way to convert the optimal strategy to an operational plan? I’m not sure if just setting target socs is enough. You would need things like “pro grid” or “pro battery” to handle unexpected loads/PV. I haven’t found a direct way to set those modes.
Hi @bartm, thanks for your proposal to help out if needed; maybe I’ll need it in the future!
You are right that a small forecasting window together with a large battery capacity leads to undesired behavior. Or, as you described it, “it thinks the world ends.”
I have not implemented a cost to the objective function for punishing this behavior. My hope is to mitigate this problem by making the forecasting window larger. I already have access to a proprietary EPEX predictor that gives the prices for 3 days. But I plan to replace it with Epexpredictor when I have the time. For the solar prediction, I have experimented with a custom-trained model, but the results fall short as of now. It was wishful thinking on my part that I could accurately predict solar output based solely on EPEX price instead of local weather predictions. I am looking for something else but have not found anything that suits my needs yet. If undesirable selling turns out to be an issue, I will look into adding a cost incentive to the objective function.
The consumption prediction is just my average base load, excluding all the loads that I predict later. The EV is a good example. I know when the EV is plugged in, and that is in solar mode. In this case I do not have to do anything to accommodate the EV; it will adjust itself. When there is a charge planned, I can add it to my existing consumption prediction and give that to the forecasting tool.
As @ndrew pointed out, the new DESS registers are not as easy to use. They have a Node RED mode where I could maybe use it, but it is uncertain if that mode will continue to exist. Instead, I have opted to try his approach first and maybe later revise it. Currently I am busy refactoring all my code into classes to finally develop the in-hour strategy. I have created an asynchronous Victron Modbus wrapper to interface with the Victron equipment, so I am quite flexible on my final solution. My hope is that I will be able to develop something that does not use the “state of charge” but instead only thinks in terms of “energy.” But for the sake of testing, I will first adapt a SoC-based strategy so I can get it tested on my system.
Here is a strategy function I had written. The idea is that every hour/delta t has x amount of energy for every strategy.
def _get_strategy(hour): # hour is just time increment (delta t)
in_hour_strategies = {}
if hour["grid_to_battery"] > 0 or hour["battery_to_grid"] > 0: # battery must be charged/discharged from/to grid to meet SoC
in_hour_strategies[0] = Strategies.TARGET_SOC
elif hour["solar_to_battery"] > 0: # solar charge hours
if hour["solar_excess"] > 0: # charge to certain extend
in_hour_strategies[0] = Strategies.PRO_BATTERY
in_hour_strategies[hour["solar_to_battery"]] = Strategies.PRO_GRID
else:
in_hour_strategies[0] = Strategies.PRO_BATTERY # charge everything in that hour
elif hour["battery_to_consumption"] > 0: # discharge to consumption is this hour
in_hour_strategies[0] = Strategies.SELF_CONSUME
in_hour_strategies[hour["battery_to_consumption"]] = Strategies.PRO_GRID
# TODO: add system to be more flexible if consumption forecast does not turn out to be correct
else:
in_hour_strategies[0] = Strategies.PRO_GRID # do not do anything use grid.
return in_hour_strategies
This allows for having self-consumption up until x amount of energy and pro-grid after that. Maybe I can add a system where really profitable hours are fully self-consume? It is frustrating when there is more consumption than forecasted and the battery stops, even though I know that full self-consumption in said hour is the best strategy.
What I’d love to see is a DESS simulator that can run various strategies from multiple people in a standardized way on a historical data set. Any ideas how to go about such an thing anybody? Where to start? Virtual VenusOS instances with virtual devices running on an accelerated clock maybe? No idea how to integrate the VRM black box scheduler though, aside reverse engineering the VRM scheduler maybe.
I have so far tested my implementation against my historical dataset. But the difficulty is that live systems are way harder to model. Forecasting uncertainty and such. But primarily, 1kwh of energy can mean that there was 1000W in 1 hour or 2000W in 0.5 hour. For a system with a 1000W inverter limit this is a big difference. If you want to model this effectively you need high resolution power history, not energy.
