밤에 쓰는 편지 밤에 쓰는 편지

* base model: GPT-2 without injecting any additional information The backbone model can be any LLM, but I used GPT-2 with 6 layers as default for simplicity.  I may conduct an ablation study on different LLM model variants and sizes. Several previous studies have demonstrated that the scaling law also applies to time-series forecasting in relation to the number of model parameters and the size o..

Previous works on time-series foundation models have shown that, to achieve good zero-shot forecasting performance, it is necessary to train on a large-scale time-series corpus that covers diverse domains, trends, seasonality patterns, and time granularities. Additionally, MOIRAI has released a large dataset as part of its efforts to create a foundation model. However, it is challenging for me t..

Several studies have trained models using synthetic time series data as a complement to real-world data or even as a standalone approach, showing comparable zero-shot performance. (ForecastPFN, TimsFM, Fu et al., 2024) Given my limited resource budget, incorporating synthetic data into my training dataset is a viable option.  Additionally, conducting an ablation study to evaluate the effectivene..

이 난관을 어떻게 타개할 것인가 batch 내에 image를 input으로 넣어주다보니, memory를 무섭게 잡아먹어서training이 지속되지 못하고 중단되는 현상이 발생한다. input image를 읽은 후에 객체를 del하고 garbage collect를 해주어도 memory 용량은 계속 줄어든다. 아마 뭔가 내가 할당된 memory를 완전히 반환하도록 하는 방법을 제대로 모르는 것 같다.그리고 설령, 그 방법을 찾더라도,이 작업을 data_loader의 batch마다 해주는 것만으로도 엄청나게 training 시간이 길어진다.  결국 memory 용량 부족으로 중단되기 전까지의 loss를 보면,학습은 잘 이루어지는 걸로 보인다.  image를 prompt에 넣어서 추가 정보를 주는 게 내 모델의..

https://arxiv.org/pdf/2310.10688https://github.com/google-research/timesfm(Oct 2023 Google Research)AbstractMotivated by recent advances in large language models for Natural Language Processing (NLP), we design a time-series foundation model for forecasting whose out-of-the-box zero-shot performance on a variety of public datasets comes close to the accuracy of state-of-the-art supervised foreca..

이상한 걸 하나 만들었다.... 지금 돌아가고 있다.... 엄청 에러 내고  수정하고수정하고또 수정한 후에.. 드디어.. 돌아간다. ㅜㅜ  텐서 차원 맞춰주는 거.. 저만 빡센가요....? transpose, permute, rearrange, squeeze, unsqueeze의 향연.... ;;;;  차라리 명시적으로 debugging mode를 코드에 넣어주고, debugging mode 분기 탈 때, layer마다 tensor shape을 print out해주는 게 확실한 거 같아.  ㅋㅋㅋ 잠깐 편의점 가서 간식 사와야 겠다. 설마 그 사이에 에러내지 않겠지......??  오오 신이시여... 부디 아름답게 마무리하도록 해주소서

https://research.google/blog/a-decoder-only-foundation-model-for-time-series-forecasting/(February 2, 2024 Google Research)TimesFM is a forecasting model, pre-trained on a large time-series corpus of 100 billion real world time-points, that displays impressive zero-shot performance on a variety of public benchmarks from different domains and granularities.Time-series forecasting is ubiquitous in..

https://arxiv.org/pdf/2310.08278https://github.com/time-series-foundation-models/lag-llama(Oct 2023)AbstractOver the past years, foundation models have caused a paradigm shift in machine learning due to their unprecedented capabilities for zero-shot and few-shot generalization. However, despite the success of foundation models in modalities such as natural language processing and computer vision..