How to get all included files used by batchcmds:compile(file.cpp) command to fill a batchcmds:add_depfiles(h1.hpp, h2.hpp....) ?

[PierreEVEN]

Hi,

I need some help to understand depfiles and batchcmds.

I'm building an UnrealEngine-like header tool that generates reflection headers.
Example :
I create a class named foo inside a file foo.hpp, then my tool will parse this file, then generate a class in 2 generated files foo.gen.cpp and foo.gen.hpp

To achieve this, I'm using a custom xmake rule that will compile the generated .gen.cpp once updated :

rule("generated_cpp", function (rule)
    set_extensions(".hpp")
    before_buildcmd_file(function (target, batchcmds, source_header, opt)

        -- Guess generated source file path
        local generated_path = string.sub(os.projectdir().."/"..source_header, string.len(target:scriptdir()) + 2)
        -- replace .hpp extension with .gen.cpp
        local generated_source = target:autogendir().."/"..string.sub(generated_path, 1, string.len(generated_path) - 3).."gen.cpp"
        -- generated classes are always private
        generated_source = generated_source:gsub("public", "private", 1)
        
        -- check if the header tool generated reflection data for our input file
        if (os.exists(generated_source)) then
            local test_str = tostring(generated_source)
            local objectfile = target:objectfile(generated_source)
            --batchcmds:vrunv(TODO : make the header tool works for one header at a time)
            batchcmds:compile(generated_source, objectfile)

            batchcmds:show_progress(opt.progress, "${color.build.object}Compiling.reflection %s", generated_source)

            batchcmds:add_depfiles(source_header) -- The problem is here (explanations bellow)

            batchcmds:set_depmtime(os.mtime(objectfile))
            batchcmds:set_depcache(target:dependfile(objectfile))
        end
    end)
end)

My problem occurs when the input header file includes other headers : xmake will not track updates of subdependencies
example :

// foo.hpp
#include "foo.gen.hpp"

class foo {}

// bar.hpp
#include "foo.hpp" // if I update "foo.hpp", "bar.gen.cpp" will not be recompiled
#include "bar.gen.hpp"

class bar : public foo {}

Using the previous rule, xmake won't be able to know when foo.hpp is updated to recompile bar.hpp because batchcmds:add_depfiles() only includes bar.hpp

However, I've seen that xmake is able to parse includes somehow and build a complete recursive list of dependencies for each class, so I guess there is a way to solve my problem by adding the list of all dependencies recursively included by my source file in batchcmds:add_depfiles().
But for now I have no clue how to do it.

Thanks in advance for your help !

[waruqi]

xmake/xmake/rules/swig/xmake.lua

Line 119 in 46b9951

import("build_module_file").swig_build_file(target, sourcefile, table.join({sourcekind = "cc"}, opt))

xmake/xmake/rules/swig/build_module_file.lua

Lines 163 to 187 in 46b9951

	if not depend.is_changed(dependinfo, {lastmtime = os.mtime(objectfile),
	values = argv
	}) then
	return
	end
	progress.show(opt.progress, "${color.build.object}compiling.swig.%s %s", moduletype, sourcefile)
	os.mkdir(path.directory(sourcefile_cx))
	-- gen swig depend file , same with gcc .d
	local swigdep = os.tmpfile()
	local argv2 = {"-MMD", "-MF", swigdep}
	table.join2(argv2, argv)
	-- swig generate file and depend file
	os.vrunv(swig.program, argv2)
	compiler.compile(sourcefile_cx, objectfile, {target = target})
	-- update depend file
	local deps = io.readfile(swigdep, {continuation = "\\"})
	os.tryrm(swigdep)
	dependinfo.files = {sourcefile}
	dependinfo.depfiles_gcc = deps
	dependinfo.values = argv
	depend.save(dependinfo, target:dependfile(objectfile))

xmake/xmake/modules/private/action/build/object.lua

Lines 91 to 114 in 46b9951

	assert(compinst:compile(sourcefile, objectfile, {dependinfo = dependinfo, compflags = compflags}))
	-- update files and values to the depfiles
	dependinfo.values = depvalues
	table.insert(dependinfo.files, sourcefile)
	-- add precompiled header to the depfiles when building sourcefile
	local build_pch
	local pcxxoutputfile = target:pcoutputfile("cxx")
	local pcoutputfile = target:pcoutputfile("c")
	if pcxxoutputfile or pcoutputfile then
	-- https://github.com/xmake-io/xmake/issues/3988
	local extension = path.extension(sourcefile)
	if (extension:startswith(".h") or extension == ".inl") then
	build_pch = true
	end
	end
	if target:has_sourcekind("cxx") and pcxxoutputfile and not build_pch then
	table.insert(dependinfo.files, pcxxoutputfile)
	end
	if target:has_sourcekind("cc") and pcoutputfile and not build_pch then
	table.insert(dependinfo.files, pcoutputfile)
	end
	depend.save(dependinfo, dependfile)

[PierreEVEN]

Thanks for the hint !
I'm not sure if I understood correctly, but if I provide a dependinfo object to compile(), it should fill the required dependency list ?
I've tried using batchcmds:compile but it doesn't do anything to the dependinfo object.
Also I don't know how to get the sourcekind in my case (opt is empty in a before_buildcmd_file callback) so I don't know how to retrieve a compinst like in your example. I assume batchcmds:compile do the same job, but I may be wrong ?

(I'm not familiar with lua at all, so I may have missed something obvious ?)

[PierreEVEN]

Update : I finally figured out how to use the compiler instance ! 🥳

import("core.tool.compiler")
local compinst = compiler.load("cxx", {target = target})
local compflags = compinst:compflags({target = target, sourcefile = generated_source, configs = opt.configs})
assert(compinst:compile(generated_source, objectfile, {dependinfo = dependinfo, compflags = compflags}))

So that worked, now I just need a way to find a way to make it works seamlessly with my batchcmd 👍

[PierreEVEN]

Yay found the solution 🥳
Thanks @waruqi !

rule("generated_cpp", function (rule)
    set_extensions(".hpp")
    before_buildcmd_file(function (target, batchcmds, source_header, opt)

        -- Guess generated source file path
        local generated_path = string.sub(os.projectdir().."/"..source_header, string.len(target:scriptdir()) + 2)
        -- replace .hpp extension with .gen.cpp
        local generated_source = target:autogendir().."/"..string.sub(generated_path, 1, string.len(generated_path) - 3).."gen.cpp"

        -- generated classes are always private
        generated_source = generated_source:gsub("public", "private", 1)
        
        if (os.exists(generated_source)) then
            local objectfile = target:objectfile(generated_source)
            --batchcmds:vrunv(TODO : make the header tool works for one header at a time)

            import("core.tool.compiler")
            import("core.project.depend")

            local compinst = compiler.load("cxx", {target = target})
            local compflags = compinst:compflags({target = target, sourcefile = generated_source, configs = opt.configs})


            local dependfile = target:dependfile(objectfile)
            -- Load existing dep infos (or create if not exists)
            local dependinfo = target:is_rebuilt() and {} or (depend.load(dependfile, {target = target}) or {})

            local depvalues = {compinst:program(), compflags}
            local lastmtime = os.isfile(objectfile) and os.mtime(objectfile) or 0

            -- Test if source file was modified
            if not depend.is_changed(dependinfo, {lastmtime = lastmtime, values = depvalues}) then
                return
            end

            batchcmds:show_progress(opt.progress, "${color.build.object}Compiling.Reflection %s", generated_source)
            -- Compile and fill the dependency list into dependinfo
            assert(compinst:compile(generated_source, objectfile, {dependinfo = dependinfo, compflags = compflags}))
            
            -- store build depvalues to detect depvalues changes
            dependinfo.values = depvalues
            depend.save(dependinfo, dependfile)
        end
    end)
end)

[waruqi]

batchcmd is imperative, and is usually used in project generators, such as xpack and xmake project -k cmake/compile_commands/makefile/.... It cannot maintain state and does not support parallelism.

If it is some simple rules, you can use batchcmd to simplify the configuration. If it is a more complex rule, it is recommended to use before_build_file instead of before_buildcmd_file.

It is more flexible and supports parallelism.

Of course, before_build_file/before_buildcmd_file can be configured at the same time, and xmake build will give priority to the former. xmake project only uses the latter.