AsciiDoc filters generate images on disk (even when embedded)

[dagwieers]

Currently AsciiDoc filters generate images on disk, but for flat XML files and HTML files that use embedded images, this is unneeded and undesirable. It would be nice if AsciiDoc could either prevent any files to be written on disk, but instead use base64 immediately, or instead clean up the files on disk after embedding.

[elextr]

On 14 February 2012 23:43, Dag Wieërs
[email protected]
wrote:

Currently AsciiDoc filters generate images on disk, but for flat XML files and HTML files that use embedded images, this is unneeded and undesirable. It would be nice if AsciiDoc could either prevent any files to be written on disk, but instead use base64 immediately, or instead clean up the files on disk after embedding.

Asciidoc does not know what a filter does or when its output may be
used, this is in the filter configuration. Filter output may not be
used until later parts of the toolchain. So there is no way for
asciidoc to know when it is safe to delete files or even what files to
delete, and the filter which does know is long gone. And not all
toolchains support embedded images so support for files is still
needed.

---

Reply to this email directly or view it on GitHub:
#29

[dagwieers]

Welcome back !

Yes, I don't want to get rid of support for files on disk, but currently when you have inline ascii-diagrams, resulting in embedded diagrams. Your directory is full of unwanted images, and I think we should find a way to fix that.

[elextr]

On 15 February 2012 01:14, Dag Wieërs
[email protected]
wrote:

Welcome back !

Yes, I don't want to get rid of support for files on disk, but currently when you have inline ascii-diagrams, resulting in embedded diagrams. Your directory is full of unwanted images, and I think we should find a way to fix that.

Well, to delete "unwanted" images we need to know the definition of an
unwanted image.

Since the filter's images go in the same directory as user supplied
images we can't just delete all image files.

Somehow a2x has to find out:

1. the image files that are generated as part of processing the
   asciidoc file (since they can be re-created)
2. which of those files the backend embedded

If a file meets both requirements then they are no longer needed and
can be deleted.

Filters can probably be made to write a list of files they generate
for item 1. but how do we get backends to say which ones are embedded?

Note that you can put images in a subdirectory so they don't clutter
the document directory.

Cheers
Lex

---

Reply to this email directly or view it on GitHub:
#29 (comment)

[dagwieers]

You are now specifically discussing a2x, while I was referring to asciidoc.

But regardless of how it is implemented today, let's think about how it would look if we would implement it with the requirement to support embedded images. I can imagine a mechanism where asciidoc instructs the filter to get base64, or the image binary data, rather than a reference to a file. So if uri-data is enabled, the filters behave differently in that respect. Of course asciidoc would also need to make a clear distinction between embedded image blocks and normal image blocks. The current way of doing that really clutters up the backend file. If only I could reclaim the hours spent ;-)

[elextr]

On 15 February 2012 19:40, Dag Wieërs
[email protected]
wrote:

You are now specifically discussing a2x, while I was referring to asciidoc.

But regardless of how it is implemented today, let's think about how it would look if we would implement it with the requirement to support embedded images. I can imagine a mechanism where asciidoc instructs the filter to get base64, or the image binary data, rather than a reference to a file. So if uri-data is enabled, the filters behave differently in that respect. Of course asciidoc would also need to make a clear distinction between embedded image blocks and normal image blocks. The current way of doing that really clutters up the backend file. If only I could reclaim the hours spent ;-)

Well asciidoc actually expects filters to pass the output directly to
them, its the filters themselves that save stuff to files. For
example the source-highlight filter doesn't leave a file hanging
around because source-highlight and pygmentize both send their output
to stdout.

"All" you have to do is re-write all filters to return base64 encoded
text via stdout if the data-uri attribute is set, otherwise to leave
it in files for the backend toolchains to use, easy :)

But since filters are external plugins they are going to do what they
want, so you can only control the four built-in ones.

Hence my suggestion that you may have to use a2x to clean up the mess
after the toolchain has been run, although as I said it isn't that
easy.

---

Reply to this email directly or view it on GitHub:
#29 (comment)

[dagwieers]

The mechanism I was talking about was more a standard interface plugins should adhere to by the AsciiDoc project. Do you think it is worthwhile to discuss on the AsciiDoc mailinglist ?

Also, the a2x currently has a problem slightly related to this, when parsing README.odt:

[dag@moria asciidoc-odf]$ a2x -v -b odt --backend-opts="--schema=relaxng/OpenDocument-v1.2-os-schema.rng" -a icons -a iconsdir=/etc/asciidoc/images/icons README.asciidoc
-snip-
asciidoc: README.asciidoc: line 248: filtering: "/etc/asciidoc/filters/ditaa/ditaa2img.py" -v -o "/tmp/tmpYmkr3D/README__1.png" --scale 4 -
Runing filter script /etc/asciidoc/filters/ditaa/ditaa2img.py
Temporary input file is /tmp/tmps3Qth9
Exec: java -jar "/etc/asciidoc/filters/ditaa/ditaa0_9.jar" "/tmp/tmps3Qth9" "/tmp/tmpYmkr3D/README__1.png" --overwrite -v --scale 4.000000 >&2

ditaa version 0.9, Copyright (C) 2004--2009  Efstathios (Stathis) Sideris

Running with options:
overwrite
verbose
scale = 4.000000
Reading file: /tmp/tmps3Qth9
Locale: en_US
Dialog.bold
Rendering to file: /tmp/tmpYmkr3D/README__1.png
Done in 1sec
asciidoc: README.asciidoc: line 248: evaluating: {counter2:image-number}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.abspath("README__1.png")}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 296: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 343: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 367: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 398: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 419: evaluating: {counter:admonition}

Traceback (most recent call last):
  File "/usr/bin/a2x", line 934, in <module>
    a2x.execute()
  File "/usr/bin/a2x", line 356, in execute
    self.to_backend()
  File "/usr/bin/a2x", line 608, in to_backend
    eval('to_%s(self)' % self.backend)
  File "<string>", line 1, in <module>
  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 87, in to_odt
    a.make_archive(opts.base_doc, self.asciidoc_file, odf_file)
  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 57, in make_archive
    shutil.copyfile(mc.group(1), os.path.join(td, mc.group(2)))
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/dag/home-made/asciidoc-odf/README__1.png'

[elextr]

[...]

On 16 February 2012 06:21, Dag Wieërs
[email protected]
wrote:

The mechanism I was talking about was more a standard interface plugins should adhere to by the AsciiDoc project. Do you think it is worthwhile to discuss on the AsciiDoc mailinglist ?

Can't hurt, but remember that filters are external programs and
asciidoc would have to continue to support the current interface (even
if deprecated) for some time.

Currently filters don't know if their output is going to be linked or
embedded, so it would be a big change for them to all be written to be
able to save a file or send to asciidoc depending on some option,
especially if they depend on some external program like ditaa.

Also, the a2x currently has a problem slightly related to this, when parsing README.odt:

---

[dag@moria asciidoc-odf]$ a2x -v -b odt --backend-opts="--schema=relaxng/OpenDocument-v1.2-os-schema.rng" -a icons -a iconsdir=/etc/asciidoc/images/icons README.asciidoc
-snip-
asciidoc: README.asciidoc: line 248: filtering: "/etc/asciidoc/filters/ditaa/ditaa2img.py" -v -o "/tmp/tmpYmkr3D/README__1.png" --scale 4 -
Runing filter script /etc/asciidoc/filters/ditaa/ditaa2img.py
Temporary input file is /tmp/tmps3Qth9
Exec: java -jar "/etc/asciidoc/filters/ditaa/ditaa0_9.jar" "/tmp/tmps3Qth9" "/tmp/tmpYmkr3D/README__1.png" --overwrite -v --scale 4.000000 >&2

ditaa version 0.9, Copyright (C) 2004--2009  Efstathios (Stathis) Sideris

Running with options:
overwrite
verbose
scale = 4.000000
Reading file: /tmp/tmps3Qth9
Locale: en_US
Dialog.bold
Rendering to file: /tmp/tmpYmkr3D/README__1.png
Done in 1sec
asciidoc: README.asciidoc: line 248: evaluating: {counter2:image-number}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.abspath("README__1.png")}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 296: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 343: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 367: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 398: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 419: evaluating: {counter:admonition}

Traceback (most recent call last):
 File "/usr/bin/a2x", line 934, in
   a2x.execute()
 File "/usr/bin/a2x", line 356, in execute
   self.to_backend()
 File "/usr/bin/a2x", line 608, in to_backend
   eval('to_%s(self)' % self.backend)
 File "", line 1, in
 File "/etc/asciidoc/backends/odt/a2x-backend.py", line 87, in to_odt
   a.make_archive(opts.base_doc, self.asciidoc_file, odf_file)
 File "/etc/asciidoc/backends/odt/a2x-backend.py", line 57, in make_archive
   shutil.copyfile(mc.group(1), os.path.join(td, mc.group(2)))
 File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
   with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/dag/home-made/asciidoc-odf/README__1.png'

The proper path wasn't being recorded, should be fixed in git
(remember I can't test ATM)

Cheers
Lex

---

---

Reply to this email directly or view it on GitHub:
#29 (comment)