밤에 쓰는 편지 밤에 쓰는 편지

생각보다 성능이 잘 나와서 사실 다소 놀라웠다. 하지만 이걸 단순히 성능적으로 우수하다고 성급하게 단정지을 수는 없다. foundation model들은 generalization을 목적으로 any task, any context length, any prediction horizon을 지원하는데 초점을 두고 마치 LLM처럼 대량의 학습을 시켜서 versitle하게 만든 후에 downstream task에 바로 가져다 쓸 수 있게 size 별로 제공을 하고 있다.  이에 비하면 나의 모델은 flexibility가 떨어진다고 할 수 있다. 만약 benchmark 성능만을 목적으로 한다면한 놈만 패는 specialist들, transformer based model 혹은 심지어 MLP-based, linea..

https://arxiv.org/pdf/2407.07726(July 2024)PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks. We evaluate PaliGemma on almost 40 diverse tasks incl..

* In-Distribution Forecasting To ensure a fair comparison, we adhere to the experimental setting in Time-MoE - fine-tune the pre-trained models on the train split of each benchmark dataset only one epoch.  ※ we need to compare with the LLM-based models! (will update soon) ETTh1 / ETTh2 96 ETTm1 / ETTm2 Weather

* Full results of Base model's zero-shot forecasting.

* The Effect of Incorporating Synthetic Data: Due to resource limitations, it is challenging to systematically evaluate the effect of training with synthetic data. In our experiments, we trained the model for only one epoch and used a small training corpus, which restricts the scope of our analysis. The primary goal of incorporating synthetically generated time series data into the training corp..

https://arxiv.org/pdf/2106.13884(Jun 2021 NeurIPS 2021)Abstract When trained at sufficient scale, auto-regressive language models exhibit the notable ability to learn a new language task after being prompted with just a few examples. Here, we present a simple, yet effective, approach for transferring this few-shot learning ability to a multimodal setting (vision and language). Using aligned imag..

base model training results #1ETTh1 / ETTh2 zero-shotETTm1 zero-shotETTm2 zero-shotWeatherTraffic

(Mar 2023)https://arxiv.org/pdf/2303.06053https://github.com/google-research/google-research/tree/master/tsmixerAbstractReal-world time-series datasets are often multivariate with complex dynamics. To capture this complexity, high capacity architectures like recurrent- or attention-based sequential deep learning models have become popular. However, recent work demonstrates that simple univariate..