ECE-ILAB-PRYMMAL/ILAB-Merging-3B-V2

TriomphantV2-ECE-78B-SLERP-V1

This model has been produced by:

• ROBERGE Marial, engineering student at French Engineering School ECE
• ESCRIVA Mathis, engineering student at French Engineering School ECE
• LALAIN Youri, engineering student at French Engineering School ECE
• RAGE LILIAN, engineering student at French Engineering School ECE
• HUVELLE Baptiste, engineering student at French Engineering School ECE

Under the supervision of:

• Andre-Louis Rochet, Lecturer at ECE & Co-Founder of TW3 Partners
• Paul Lemaistre, CTO of TW3 Partners

With the contribution of:

• ECE engineering school as sponsor and financial contributor
• François STEPHAN as director of ECE
• Gérard REUS as acting director of iLAB
• Matthieu JOLLARD ECE Alumni
• Louis GARCIA ECE Alumni

Models Merged

The following models were included in the merge:

• microsoft/Phi-3.5-mini-instruct
• meditsolutions/MedIT-Mesh-3B-Instruct

Configuration

The following YAML configuration was used to produce this model:

slices:
  - sources:
      - model: meditsolutions/MedIT-Mesh-3B-Instruct
        layer_range: [0, 32]
      - model: microsoft/Phi-3.5-mini-instruct
        layer_range: [0, 32]    
merge_method: slerp
base_model: meditsolutions/MedIT-Mesh-3B-Instruct
parameters:
  t:
    - filter: self_attn
      value: [0, 0.25, 0.5, 0.75, 1]
    - filter: mlp
      value: [1, 0.75, 0.5, 0.25, 0]
    - value: 0.5
dtype: bfloat16