onnx-community

/

Phi-3.5-mini-instruct-onnx-web

like 10

In this case, you need to specify the correct dtype and device (it's a special optimized model for WebGPU). The following should work:

import { pipeline, env } from "https://cdn.jsdelivr.net/npm/@huggingface/[email protected]";

const languageModel = await pipeline('text-generation', 'onnx-community/Phi-3.5-mini-instruct-onnx-web', {
  dtype: 'q4f16', device: 'webgpu'
});

We are in the process of adding default values to the config which would mean you don't need to do this in future. Hope this helps!

Thanks very much for the response @Xenova ! Appreciate that pointer, it's helped me get further! As an aside - I've been so pleasantly surprised by how willing people are to help here :-)

I've got it loading, I still seem to be hitting an error with

I seem to be hitting this error now:

Error: Can't create a session. ERROR_CODE: 1, ERROR_MESSAGE: Deserialize tensor model.layers.4.attn.o_proj.MatMul.weight_Q4 failed.Failed to load external data file ""model_q4f16.onnx_data"", error: Module.MountedFiles is not available.
    at Ve ([email protected]:100:73169)
    at Zu ([email protected]:100:357172)

It appears with these warnings, but I don't think they prevent the model from running:

1.

2024-10-01 09:28:44.905599 [W:onnxruntime:, session_state.cc:1168 VerifyEachNodeIsAssignedToAnEp] Some nodes were not assigned to the preferred execution providers which may or may not have an negative impact on performance. e.g. ORT explicitly assigns shape related ops to CPU to improve perf.

2.

2024-10-01 09:28:44.906799 [W:onnxruntime:, session_state.cc:1170 VerifyEachNodeIsAssignedToAnEp] Rerunning with verbose output on a non-minimal build will show node assignments.

In case it helps, the below is my full code - I'm running on a Macbook Pro in Chrome (exact same setup that I used to try out the Phi demo). I initially tried just running the code locally with VS Code Live Server but just in case that was the culprit I also uploaded to my server and tried loading the page there but got the same errors

const status = document.getElementById("status");
status.textContent = "Loading model...";
import { pipeline, env } from "https://cdn.jsdelivr.net/npm/@huggingface/[email protected]";

try {  
  console.log("Started model loading")
  const languageModel = await pipeline('text-generation', 'onnx-community/Phi-3.5-mini-instruct-onnx-web', {
    dtype: 'q4f16', device: 'webgpu'
  });
  console.log("Finished model loading")
  status.textContent = "Ready";
  generateText(languageModel);
} catch (err) {
  console.log(err)
  status.textContent = "Error - failed to load";
}

Thanks again for the help!

Made a quick edit to the above - realised I was doing something silly with my error handling (eesh). I have updated with the actual error I'm getting!

Never seen that one before 0_0

Oh whoops - you also need to add use_external_data_format: true as an option:

const languageModel = await pipeline('text-generation', 'onnx-community/Phi-3.5-mini-instruct-onnx-web', {
  dtype: 'q4f16', device: 'webgpu', use_external_data_format: true,
});

Hi there! Sorry for the delay in response - I wanted to double-check before I came back again. It looks like I'm getting the same kind of odd error.

I think I'm going to end up building what I planned in Angular anyway, for unrelated reasons, so I think I should be able to use the npm version of this library which may behave differently so I'll give that a crack.

Thanks very much for the time and effort to help! Sorry I can't report more success with this method!