Machine learning (ML) and artificial intelligence (AI) present some fascinating modeling approaches. However, even with ML and hyper-fast computing, the old saying "garbage in, garbage out" or simply GIGO holds true. EQ provides structured and curated data for modeling. Our database is a treasure trove for model developers.
EQ = self-service
Help yourself to data. EQ’s business is to provide the best possible curated historical data and aligned forecasts, empowering you to excel in your job with minimum waste of time and energy. If you don't already have an account, get your access here.
Looking for something that is not there
Throwing in more data will not help you (much) if what you are looking for is not there. I have tried to illustrate the problem with two pictures. In the left picture, I have simply removed information and added noise, analog to a what you might see in a dataset.
You can easily observe the problem in a picture. However, in data analysis, you can, at best, anticipate that something is missing. It could be too big times steps, poor spatial resolution, misaligned input used for model fitting and forecasting, or good old fashioned out-layers and missing values. More data might add to the problem and certainly waste your time.
A checklist
A checklist before you hit enter "estimate model" might save you a lot of time:
- Check your input data. At EQ, we do this as a profession. For instance, we provide curated synthetic data which we believe is a far superior alternative to actuals generated by TSOs.
- Avoid settling for raw weather data. Failing to align the output generated by weather models with real-life observations creates nasty errors and gut-wrenching frustrations. I know, from personal experience.
- Consider adding contextual information. Capacity changes, holiday effects, REMIT data, and more input add depth to the analysis. Failing to add context might contaminate your model and drain it for explanatory power.
- Look out for "known" non-linearity. A simple trick is, e.g. to use transformed data. Regardless of how advanced your model is, this simple trick might limit the data requirement tremendously, and improve the forecasts properties.
Take vitamins, not medicine
EQ provides a vast set of curated historical weather-driven fundamentals and indices. We believe they are perfect input for modeling and forecasting. Further, we provide all data in 15 minutes resolution, which is likely to become the power market’s preferred resolution. Using the correct data granularity from the get-go hugely simplifies modeling.
Contact me to discuss the best solutions for your team. We may have your data requirement covered, faster and simpler than you can imagine.
...I almost forgot
You do not have to download EQ’s data. Connect your developer tool or models directly into EQ’s timeseries database. It saves you time and simplifies your life.
EQ generates approximately 4,5 million data points per price areas every day. It makes sense to leave the data in our custody.
