[breaktime] LLM's pattern recognition

Just for fun!

 

I'm just curious about how can model recognize the patterns in the sequences.

Later on, I'm planning to research on how the model with mixture of attention and s6 layer can recoginize various patterns in the sequences. I think it will be interesting.

 

I trained my base model* (gpt2 frozen, wo/ additional information injection, only alignment(cross-attn) & head parameters updated) on 10% of synthetic dataset (3 epochs) and investigated the zero-shot performance on ETTh1.

---

Args in experiment:
Namespace(is_training=0, model_id='512_96', model='MyModel', seed=2021, data='ETTh1', target_data='ETTh1.csv', root_path='./dataset/ETT-small/', data_path='ETTh1.csv', checkpoints='./checkpoints/', seq_len=512, pred_len=96, enc_in=7, d_model=32, n_heads=8, d_ff=128, dropout=0.1, patch_len=16, stride=8, llm_dim=768, num_workers=0, itr=1, train_epochs=3, batch_size=256, patience=10, learning_rate=0.001, lradj='type1', pct_start=0.2, llm_layers=6)

 

>>>>>>>testing : 512_96_MyModel_Synthetic_sl512_pl96_dm32_nh8_df128_0<<<<<<

self.enc_in = 7
self.data_x = (3392, 7)
loading model
test on the ETTh1.csv dataset: mse:1.9248285293579102, mae:0.8418431282043457

 

TimesFM zero-shot

 

MOIRAI zero-shot

---

< Training on synthetic dataset visualization > 

이렇게 당황스럽게 못하기도 합니다. 하하 ;;

 

---

< zero-shot prediction visualization >