Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022
•
0 likes•133 views
Time-series data is all around us: from logistics to digital marketing, from pricing to stock markets - it’s hard to imagine a modern business that has no time series data to forecast. However, mastering such forecasting is not an easy task. For this talk, we have collected a list of common time series issues that digital fortune tellers commonly run into. You will learn how to identify, understand and resolve them better. This will include stabilising divergent time series, handling outliers without anomaly propagation, reducing the impact of noise and more.
Report
Share
More Related Content
Similar to Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022
Similar to Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022 (20)
More from GoDataDriven
More from GoDataDriven (20)
Recently uploaded
Recently uploaded (20)
Common Issues With Time Series by Vadim Nelidov - GoDataFest 2022
- 1. Common Issues with Time Series and how to solve them Vadim Nelidov GoDataFest 2022
- 2. About me • Data Science consultant @GoDataDriven since before Corona • Background in Statistics & Econometrics • Favorite applications / projects: o Time Series o Probabilistic modelling o Image recognition o Recommenders • Collecting (fun) facts about ants, cats and astrophysics
- 3. Time Series: why you should care • Any regularly collected data is a potential TS problem • TS Forecasting is often business critical (sales, profits, consumption, stock) • TS data specifics are often misunderstood and downplayed • Majority of our consultants reported clients having issues with Time Series data (if such data was present)
- 4. Today: • What can go wrong with energy demand forecasting? • Overview of common & interesting TS problems o taming delayed data o stabilizing divergent time series o handling difficult missing values & intervals o reducing the impact of noise • Discussing effective ways to solve them (with code) • Lots of time series plots • Q&A along the way
- 5. TS Problems are not just about modeling pulling relevant data together Inspection, analytics & finding issues solving these issues actual time series modeling
- 6. 1. Delayed & irregular data Symptoms: some features are only available with a lag / time gaps Common cause: complex collection processes for some data / manually collected data / governmental sources etc. Goal: identify matching temperature for each date
- 7. 1. Delayed & irregular data Bad Solution: make intuitive assumptions and manually shift data around Assumption above: temperatures are always collected with a 1-day lag à shift them by 1 row à can you see an issue there?
- 8. 1. Delayed & irregular data Better Solution: explicitly merge data based on available timestamps this approach needs no assumptions and handles rare exceptions correctly Tip: make sure all your data has timestamps for each step of its lifecycle
- 9. 1. Delayed & irregular data Solution in code:
- 10. pulling relevant data together Inspection, analytics & finding issues solving these issues actual time series modeling 2. Unstable time series dynamics
- 11. 2. Unstable time series dynamics Symptoms: expanding / shrinking variance; dynamic trends energy usage over time
- 12. 2. Unstable time series dynamics Common cause: underlying (business) variables growing / shrinking overtime number of customers over time
- 13. 2. Unstable time series dynamics Solution: divide TS by this underlying variable and forecast the new TS instead energy usage per customer over time Advantages: more stability; separates usage and customer numbers forecasting problems
- 14. pulling relevant data together Inspection, analytics & finding issues solving these issues actual time series modeling 3. Difficult missing values
- 15. 3. Difficult missing values Symptoms: missing values near local anomalies / clustered in time Common cause: data collection issues / temporal anomalies / naturally irregular data
- 16. Solution: use (exponentially) smoothed values as a replacement 3. Difficult missing values
- 17. 3. Difficult missing values Solution in code:
- 18. pulling relevant data together Inspection, analytics & finding issues solving these issues actual time series modeling 4. Noise and outliers hindering forecasting
- 19. 4. Noise and outliers Symptoms: forecasting models pick up too much irregularities & overfit Common cause: irrelevant noise & outliers in training data, while models are punished for missing them
- 20. 4. Noise and outliers Solution: train, optimize and evaluate models with smoothed time series Now both the model & its evaluator focus on what we actually care about (prev model would have 23% here)
- 21. 4. Noise and outliers Solution in code:
- 22. Take Aways • Time Series are Special! Treat them accordingly • The way you piece your data together has long lasting impact • Careful inspection of your data & analytics pay off • Behind divergent TS are often some other TS • Careless imputation may propagate anomalies and other issues • The best solution to a problem is often to reconsider it
- 23. Questions? and thank you for being here!