Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try to apply softmax to a batch of data with variable length #1297

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

hfxunlp
Copy link
Contributor

@hfxunlp hfxunlp commented Nov 20, 2017

Hi, I want to make SoftMax support variable length input, so you can use a batch of data with different length as the input of this module. This is helpful for Natural Language Processing, especially for the Attention model of seq2seq and Attention-over-Attention model for reading comprehension.
I have also raised a corresponding pull request at torch/cunn#489.

@hfxunlp
Copy link
Contributor Author

hfxunlp commented Nov 20, 2017

I appreciate if I can get any chance to learn how to contribute to THNN, but I have not found any documents currently. Sorry for my lack of skills.

@hfxunlp
Copy link
Contributor Author

hfxunlp commented Nov 21, 2017

Sorry for I miss the declaration in lib/THNN/generic/THNN.h, and I think these code can work correctly now.

@hfxunlp
Copy link
Contributor Author

hfxunlp commented Nov 21, 2017

There is the test script that I used to check whether this patch can work correctly or not:

require "nn"
tmodstd=nn.SoftMax()
tmod=nn.LenSoftMax()
minbsize=20
maxbsize=100
minlen=16
maxlen=128
minpadlen=4
maxpadlen=16
psg=true
firstcycle=100
for t=1, firstcycle do
	if psg then
		bsize=math.random(minbsize, maxbsize)
		lens=math.random(minlen, maxlen)
		plens=math.random(minpadlen, maxpadlen)
		lvec=torch.LongTensor(bsize):fill(lens)
		stdi=torch.randn(bsize, lens)
		i=torch.cat(stdi, torch.randn(bsize, plens))
		stdgo=torch.randn(bsize, lens)
		go=torch.cat(stdgo, torch.randn(bsize, plens))
		stdo=tmodstd:forward(stdi)
		o=tmod:forward({i, lvec})
		if not (o:narrow(2, 1, lens):equal(stdo) and o:narrow(2, lens+1, plens):equal(torch.zeros(bsize, plens)) ) then
			psg=false
			print("forward error")
		end
		stdgi=tmodstd:backward(stdi, stdgo)
		gi=tmod:backward({i, lvec}, go)
		if not (gi:narrow(2, 1, lens):equal(stdgi) and gi:narrow(2, lens+1, plens):equal(torch.zeros(bsize, plens)) ) then
			psg=false
			print("backward error")
		end
	end
	xlua.progress(t, firstcycle)
end
if psg then
	print("test pass")
end

@hfxunlp hfxunlp changed the title Try to support variable length SoftMax Try to apply softmax to a batch of data with variable length Nov 21, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants