밤에 쓰는 편지 밤에 쓰는 편지

※ 2024 NLP class team project's research subjectAbstractAdvances in large language models (LLMs) have revolutionized the natural language processing field. However, the text generated by LLMs can result in various issues, such as fake news, misinformation, and social media spam. In addition, detecting machine-generated text is becoming increasingly difficult because it produces text that resembl..

처음 의도했던 것은 univariate만을 고려하는 foundation model의 한계를 극복하고, variate 간의 complex correlation을 고려할 수 있도록 vision prompting을 넣어주는 것이었는데, 이는 input dimension이 엄청 증가한다는 것을 의미하고, 결국 구현 상의 한계를 뛰어넘지 못했다. 대신 static한 covariate 정보를 넣어주었는데, 기존에 의도한 바를 구현하는 과정에서 하도 삽질을 하고 고통을 겪어서인지, 상당히 쉽게 느껴졌다.  그러고도 결과를 크게 기대하지 않았었는데, 오히려 무거운 vision encoder와 vision projector가 붙어서, (물론 pretrained된 siglip parameter를 가져왔지만 - specif..

* The effect of injecting auxiliary information via vision prompting In this section, we assess whether incorporating exogenous information via vision prompts as a prefix helps guide the large language model (LLM) to improve forecasting accuracy. To investigate the effect of vision prompting, we evaluate the forecasting performance of our pretrained model both with and without vision prompts on ..

생각보다 성능이 잘 나와서 사실 다소 놀라웠다. 하지만 이걸 단순히 성능적으로 우수하다고 성급하게 단정지을 수는 없다. foundation model들은 generalization을 목적으로 any task, any context length, any prediction horizon을 지원하는데 초점을 두고 마치 LLM처럼 대량의 학습을 시켜서 versitle하게 만든 후에 downstream task에 바로 가져다 쓸 수 있게 size 별로 제공을 하고 있다.  이에 비하면 나의 모델은 flexibility가 떨어진다고 할 수 있다. 만약 benchmark 성능만을 목적으로 한다면한 놈만 패는 specialist들, transformer based model 혹은 심지어 MLP-based, linea..

https://arxiv.org/pdf/2407.07726(July 2024)PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly knowledgeable base model that is effective to transfer. It achieves strong performance on a wide variety of open-world tasks. We evaluate PaliGemma on almost 40 diverse tasks incl..

* In-Distribution Forecasting To ensure a fair comparison, we adhere to the experimental setting in Time-MoE - fine-tune the pre-trained models on the train split of each benchmark dataset only one epoch.  ※ we need to compare with the LLM-based models! (will update soon) ETTh1 / ETTh2 96 ETTm1 / ETTm2 Weather

* Full results of Base model's zero-shot forecasting.

* The Effect of Incorporating Synthetic Data: Due to resource limitations, it is challenging to systematically evaluate the effect of training with synthetic data. In our experiments, we trained the model for only one epoch and used a small training corpus, which restricts the scope of our analysis. The primary goal of incorporating synthetically generated time series data into the training corp..