Live2D with Lipsync (Seperated) #117

RaSan147 · 2023-10-06T12:35:49Z

No description provided.

Changes: 1. updated model.motion(group, index, priority) to model.motion(group, index, priority, sound, volume, expression); 2. added model.stopSpeaking() 3. updated readme.MD with demos 4. Workflow will save files to dist (won't be gitignored) 5. Praying this new change doesn't break

voice volume expressions are now optional arg {name: value, ....}

guansss

Sorry for the long wait! You did a great job on this PR and I really appreciate the effort you put into it.

I've added a few comments for things that need to be addressed before this can be merged.

.github/workflows/test.yml

.gitignore

guansss · 2023-12-01T10:26:19Z

README.md

+#### Feel free to support the Maintainer:
+<a href="https://www.buymeacoffee.com/RaSan147" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>
+


It's okay to put this in your fork, but not into the main repository.

Will make another fork 😅 copy pasted it in all my repo. Sorry.

guansss · 2023-12-01T10:29:25Z

README.md

 ```

-#### Via CDN
+#### Via CDN (lipsync patched)


This and the (not lipsync patch) above is unnecessary after this feature is merged.

Yeah will change that.

guansss · 2023-12-01T10:30:29Z

README.md

+<script src="https://cdn.jsdelivr.net/gh/RaSan147/[email protected]/dist/index.min.js"></script>

 <!-- if only Cubism 2.1 -->
-<script src="https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism2.min.js"></script>
+<script src="https://cdn.jsdelivr.net/gh/RaSan147/pixi-live2d-display@v0.4.0-ls-2/dist/cubism2.min.js"></script>

 <!-- if only Cubism 4 -->
-<script src="https://cdn.jsdelivr.net/npm/pixi-live2d-display/dist/cubism4.min.js"></script>
+<script src="https://cdn.jsdelivr.net/gh/RaSan147/pixi-live2d-display@v0.4.0-ls-2/dist/cubism4.min.js"></script>


These links should point to the main repository.

guansss · 2023-12-01T11:52:11Z

src/cubism-common/SoundManager.ts


 const TAG = 'SoundManager';
-const VOLUME = 0.5;
+const VOLUME = 0.9;


This should not be changed. 0.9 would be too loud as a default volume.

However since the lip movement value is dependent on the audio volume, lowering it makes unusual movement (like mumbling)

Is it possible to add a multiplier after getting the value? Some thing like amplitude *= 2

nope, like if you set volume to 0, there will be no response, no matter how the audio or multiplier is (Browser JS API, cant change).... But yeah multiplier can be really great, will try to add one.

tried, sending Amp variable to analizer is hard. Let it be...

guansss · 2023-12-01T11:56:21Z

src/cubism-common/SoundManager.ts


        audio.volume = this._volume;
        audio.preload = 'auto';
+        audio.autoplay = true;


autoplay should not be true because we are not immediately playing the audio. When config.motionSync is on, we'll postpone it until the related motion/expression is ready.

Thanks for the info

guansss · 2023-12-01T12:20:49Z

src/cubism-common/SoundManager.ts

        audio.volume = this._volume;
        audio.preload = 'auto';
+        audio.autoplay = true;
+        audio.crossOrigin = "anonymous";


crossOrigin should not be set by default, otherwise if the audio's source is cross-origin but the server does not respond with a CORS header, the audio will be blocked by the browser.

It should be an option, for example, model.speak('audio.mp3', { crossOrigin: true })

guansss · 2023-12-01T12:25:12Z

src/cubism2/Cubism2InternalModel.ts

    angleZParamIndex: number;
    bodyAngleXParamIndex: number;
    breathParamIndex: number;
+    mouthFormIndex: number;


This property seems unused.

I do remember it using somewhere in cubism 2, cubism 4 has it by default. It takes the mouthForm id...

guansss · 2023-12-01T12:30:01Z

src/cubism4/Cubism4InternalModel.ts

+    updateFacialEmotion(mouthForm: number) {
+        this.coreModel.addParameterValueById(this.idParamMouthForm, mouthForm); // -1 ~ 1
+    }


This is not related to lip sync, I suppose?

Thats the main thinggggg. 😅 It makes the lips move.

RaSan147 · 2023-12-07T12:31:25Z

Sorry for the late reply.
I'll make a branch with the changes you mentioned (also fixing a lot of merge conflict from the latest update)
Thanks for taking your time and looking into it. This project is awesome.

Co-authored-by: Guan <[email protected]>

also remove cache buster and autoplay

Fix/fix audio memory leak

to test CI success

Improve test case stability in GitHub action

fix(vite.config.ts): add missing bundleTestFiles

Fix crash when model has no clipping masks Also fixes #12

correct spell miss

Update README.md

Changelog: Improved stability Added translated readme Removed unwanted dep Fix GitHub action Updated submodule policy

RaSan147 added 9 commits October 6, 2023 18:16

Automated report

d378339

Update index.js links

3ff3a53

Update test.yml

8539e76

Update test.yml

cf0f4c4

Update test.yml

f53e25a

Fix Lip sync. Breaking change

7deba12

voice volume expressions are now optional arg {name: value, ....}

Automated report

06bfbd3

Update README.md and added video

1e89254

guansss requested changes Dec 1, 2023

View reviewed changes

RaSan147 added 12 commits December 7, 2023 05:21

Merge branch 'master' into master

34831e8

Fix type error on ci test

2d0c512

clear dist, using workflow to get output files

2617eae

removed Version number

de764a1

Now supports other b64 audio

2877f47

Remove audio url validation

25ddef6

rename speakUp -> speak

b4a394a

rename resetMotions -> stopMotions

6f3a779

return false when audio play fails

e878d79

Added crossorigin for speak voice source

031ca86

Place missing CrossOrigin arg

ae1c923

Update MotionManager.ts

c0cb7f5

RaSan147 changed the title ~~Live2D with Lipsync (using audio file/link)~~ Live2D with Lipsync (Outdated) Dec 14, 2023

Update readme as per PR

c8eca3f

RaSan147 mentioned this pull request Dec 14, 2023

Live2D with Lipsync (using audio file/link) #122

Open

RaSan147 and others added 4 commits December 14, 2023 19:05

Check any base64 data for audio

7f8ec7a

Co-authored-by: Guan <[email protected]>

Remove autoplay attr from audio

06d7255

Co-authored-by: Guan <[email protected]>

Remove cache blocker.

96a6f82

Co-authored-by: Guan <[email protected]>

remove wav only blob condition

9fc5a21

also remove cache buster and autoplay

RaSan147 and others added 23 commits March 26, 2025 04:22

removed logging onFinish & lint a bit

64f580e

Merge pull request #24 from shiftonetothree/fix/fix-audio-memory-leak

ff456f0

Fix/fix audio memory leak

Update .gitmodules

fcebcc6

to test CI success

Run with missing lib installed

d989f86

bring back

d7e85f3

Automated report

946298c

test updates

8f70eae

Update test.yml

927cec8

Revert patch-dts-generator.js

c76c4ab

Update compatibility.test.ts

0bada4a

feat(stability.test.ts): fix ci

e717250

fix(vite.config.ts): fix ci

8117fdd

feat(vite.config.ts): set testTimeout to 10s

56e52e0

Merge pull request #25 from shiftonetothree/CI-test

bd67014

Improve test case stability in GitHub action

fix(vite.config.ts): add missing bundleTestFiles

34c9527

fix(vite.config.ts): set fileParallelism to false

2de3c2a

Merge pull request #26 from shiftonetothree/CI-test

9f42253

fix(vite.config.ts): add missing bundleTestFiles

removed submodule download part

aaebece

fix(cubism4): skip updateWebGLContext when clipping manager is absent

e24c2d3

fix error of branch or tag could not be found (CI)

686d282

Automated report

38e0218

Merge pull request #27 from shinshin86/fix/cubism4-no-clipping-crash

01e14cd

Fix crash when model has no clipping masks Also fixes #12

Automated report

2d4d12d

RaSan147 changed the title ~~Live2D with Lipsync (Outdated)~~ Live2D with Lipsync (Seperated) May 2, 2025

mikeyang01 and others added 6 commits May 20, 2025 22:52

Update README.md

7c6a0e3

correct spell miss

Merge pull request #28 from mikeyang01/patch-1

c907f20

Update README.md

Update to v0.5.0-ls-8

6c5275f

Changelog: Improved stability Added translated readme Removed unwanted dep Fix GitHub action Updated submodule policy

Automated report

0d6e8ef

update submodule stuffs and build

9938a1a

Automated report

6c2645f

		#### Feel free to support the Maintainer:
		<a href="https://www.buymeacoffee.com/RaSan147" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>

Uh oh!

Live2D with Lipsync (Seperated) #117

Are you sure you want to change the base?

Live2D with Lipsync (Seperated) #117

Uh oh!

Conversation

RaSan147 commented Oct 6, 2023

Uh oh!

guansss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RaSan147 commented Dec 7, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants